- Research article
- Open Access
Sexual orientation differences in treatment expectation, alliance, and outcome among patients at risk for suicide in a public psychiatric hospital
BMC Psychiatry volume 17, Article number: 184 (2017)
Sexual minority (SM) individuals (gay, lesbian, bisexual, or otherwise nonheterosexual) are at increased risk for mental disorders and suicide and adequate mental healthcare may be life-saving. However, SM patients experience barriers in mental healthcare that have been attributed to the lack of SM-specific competencies and heterosexist attitudes and behaviors on the part of mental health professionals. Such barriers could have a negative impact on common treatment factors such as treatment expectancy or therapeutic alliance, culminating in poorer treatment outcomes for SM versus heterosexual patients. Actual empirical data from general psychiatric settings is lacking, however. Thus, comparing the treatment outcome of heterosexual and SM patients at risk for suicide was the primary aim of this study. The secondary aim was to compare treatment expectation and working alliance as two common factors.
We report on 633 patients from a suicide prevention inpatient department within a public psychiatric hospital. Most patients were at risk for suicide due to a recent suicide attempt or warning signs for suicide, usually in the context of a severe psychiatric disorder. At least one indicator of SM status was reported by 21% of patients. We assessed the treatment outcome by calculating the quantitative change in suicide ideation, hopelessness, and depression. We also ran related treatment responder analyses. Treatment expectation and working alliance were the assessed common factors.
Contrary to the primary hypothesis, SM and heterosexual patients were comparable in their improvement in suicide ideation, hopelessness, or depression, both quantitatively and in treatment responder analysis. Contrary to the secondary hypothesis, there were no significant sexual orientation differences in treatment expectation and working alliance. When adjusting for sociodemographics, diagnosis, and length of stay, some sexual orientation differences became significant, indicating that SM patients have better outcomes.
These unexpected but positive findings may be due to common factors of therapy compensating for SM-specific competencies. It may also be due to actual presence of SM competencies – though unmeasured – in the department. Replication in other treatment settings and assessment of SM-specific competencies are needed, especially in the field of suicide prevention, before these findings can be generalized.
Compared to heterosexuals, sexual minority (SM) individuals are at increased risk for mental disorders, suicide attempts, and suicide [1,2,3]. SM is an umbrella term that covers different populations of individuals who are not exclusively heterosexual. SM include individuals with nonheterosexual identity (gay, lesbian, bisexual, mostly heterosexual, queer, questioning, etc.), nonexclusive heterosexual behavior (bisexual or same-sex behavior), or nonexclusive heterosexual attraction. These subpopulations share an increased risk for mental health disorders and suicide, across gender, country, or year of study . Sexual orientation disparities are large for suicides and suicide attempts but also notable for disorders known to be associated with increased suicide risk, especially depression and substance - related disorders [4, 5]. Data on other Axis I and Axis II disorders are sparse but generally suggest an increased risk for SM individuals, too . Explanatory models about this increased risk center around general stressors and stigma-related stressors that are specific for sexual minorities: For example, such stressors include experiences and fear of discrimination and violence because of one’s sexual orientation, internalized homophobia, stress associated with hiding one’s sexual orientation, lack of social support [6,7,8,9]. The findings of recent studies support these explanatory models (e.g., [10,11,12,13,14,15]).
Do SM individuals benefit less from mental healthcare treatment?
SM individuals have been recognized as a target group for mental health intervention  and suicide prevention . However, SM patients seem to face interpersonal and structural barriers to appropriate adequate healthcare  that resemble enacted and expected stigma  within healthcare systems. On an interpersonal level, there may be experiences of discrimination, microaggressions, harsh language, rejection, denial of service, attempts to change sexual orientation, silencing sexual orientation issues, or implicit biases [16, 18,19,20,21,22,23,24]. On an intrapersonal level, there may be expectations of these behaviors, hindering disclosure to health professionals . On a structural level, healthcare providers often lack cultural competency with respect to SM and gender minority issues and experience difficulties in providing SM-affirmative healthcare [16, 24,25,26,27,28]. For example, 85% of medical students reported lacking SM-specific healthcare education across all years in a recent UK study . Given these barriers, it is not surprising that SM patients report lower levels of satisfaction with mental healthcare than heterosexuals , and remain frequently undisclosed or avoid sexual orientation issues altogether [19, 30,31,32]. Consequently, there is interest in and demand for SM-specific treatment [3, 16, 33]. Despite a growing number of institutions specializing in healthcare for SM individuals, specific interventions remain exceptions in most settings, meaning that most SM individuals will receive usual care.
Based on the personal and structural barriers described above, it can be hypothesized that SM patients may benefit less from usual mental healthcare. This is especially problematic in the case of suicidal SM patients, where appropriate treatment could be life-saving. Knowing whether SM individuals actually derive less benefit from general psychiatric treatment is crucial to understand the scope of the problem and the urgency for necessary changes in mental healthcare. Thus, the primary goal of our study is to explore whether SM and heterosexual patients at risk for suicide differ in their benefit from treatment in a setting that is not specifically tailored for SM patients.
To our knowledge, no studies have been published which assess how SM patients benefit from regular treatment in psychiatric care, including psychiatric suicide prevention measures such as inpatient crisis intervention. This gap is likely a result of the lack of systematic assessment of sexual orientation in healthcare settings. Most studies used selected samples of sexual minority individuals who reported their healthcare experiences retrospectively (e.g., [20, 22, 32, 34, 35]), and it is not possible to assess the actual impact of barriers on outcomes among SM patients from these samples. In our study, we collected information on sexual orientation from all patients in a treatment setting. Thus, selection for treatment and selection for study participation did not vary with the sexual orientation of patients, facilitating assessment of potential differences in treatment effects between SM and heterosexual patients.
Are common treatment factors less optimal for SM patients?
Beyond identifying the mere difference in treatment outcome for SM versus heterosexual patients, a secondary goal of our paper is to explore the impact of mechanisms known to be important for successful treatment by sexual orientation. In the contextual model of psychotherapy , contextual factors for treatment success have been outlined: empathic and real therapeutic relationship, creation of positive expectations, shared explanations of the problem, agreement on treatment goals, tasks, and therapeutic and healthy actions. Research has shown that a good therapeutic alliance is crucial in suicide prevention as well [37,38,39,40]. Furthermore, a positive treatment expectation and a good patient-clinician relationship are important beyond psychotherapeutic treatments, for example in pharmacotherapy, where the two factors enhance effectiveness of and adherence to the treatment of psychiatric disorders [36, 41, 42].
Given the interpersonal and structural barriers that SM patients have reported, it is plausible that common factors such as treatment expectancy and therapeutic relationship may be impaired. Some patients may fear rejection or even discrimination from mental health professionals, leading to a negative expectation that treatment would be helpful. For older patients, this expectation may stem from the long-standing pathologization of SM individuals in psychiatry. Impaired treatment expectations may also result from previous negative experience in healthcare, or via projections of negative societal experiences onto therapeutic settings. Therapeutic alliance may be influenced negatively by impaired treatment expectancy, by the lack of cultural competency of healthcare professionals, or even by verbal or nonverbal discrimination.
Based on the theoretical assumption outlined above, our primary hypothesis is that treatment outcome is less optimal for SM compared to heterosexual patients. According to the secondary hypothesis, treatment expectation and working alliance as two common factors of treatment are similarily less optimal for SM than heterosexual patients.
Participants and procedure
Between the assessment period of February 5, 2010, to February 26, 2014, 997 admissions of individual patients were recorded at the department of crisis intervention and suicide prevention (CI-SP), which is part of the public psychiatric hospital of the City of Salzburg, Austria (“Christian Doppler Clinic”). The province of Salzburg is the sole proprietor of the hospital. There is no other psychiatric hospital in the region; thus, nearly all patients who require inpatient treatment are admitted to this hospital. The CI-SP department belongs to one of five departments for patients with acute mental disorders and is specialized in suicide prevention. Adult patients (18 years or older) are selected for treatment in CI-SP department predominantly after suicide attempts or other crisis situations with warning signs of suicide, such as suicide ideation, despair, or hopelessness . Usually, patients’ suicidal crisis occurs within the context of a psychiatric disorder, most often affective disorders with a current depressive episode, adjustment disorders, and frequently comorbid with substance - related disorders or Cluster B Axis II personality disorders. There is consensus that such patients are at heightened risk for suicide . Patients with acute psychotic disorders are usually not treated since the treatment-method is talk-oriented crisis intervention (see below). Infrequently, other patients are treated at the CI-SP department, since there is a mandatory admission for all patients in Salzburg with acute mental health disorders and sometimes open beds are unavailable at other departments.
The treatment method is crisis intervention, with a planned stay of 3–14 days and with longer stays in cases of enduring major depressive episodes or other factors complicating discharge (e.g., lack of housing), or temporary transfers to the closed ward in case of imminent suicide risk. Following usual crisis intervention plans, high-frequency meetings between patient and his/her responsible psychotherapist or clinical psychologist form the modality of treatment. Crisis intervention steps are assessing suicide risk, establishing rapport, analyzing the problem in a collaborative way, managing feelings, exploring alternatives, developing a positive plan, safety planning, and follow-up . Crisis intervention is adapted for patients with borderline personality disorder by including elements of Dialectical Behavior Therapy . Additional forms of treatment are applied to support and realize crisis intervention and include medication, psychiatric nursing, occupational therapy, physiotherapy, relaxation groups, and psychoeducational groups. Since most of the patients suffer from acute mental disorders, most frequently a depressive episode, nearly all of the patients receive antidepressants. Common additional medications are antipsychotics, mood-stabilizers, and benzodiazepines. The medication is adapted in daily meetings with a psychiatrist.
At admission, the responsible psychiatrist and psychotherapist/psychologist interview the patient in a narrative style to enhance therapeutic alliance. This is followed by a structured assessment of psychiatric diagnosis, physical symptoms, suicidality (suicide ideation, plans, self-control), previous suicide attempts (date, reason, method, medical treatment), nonsuicidal self-injury, aggressive behavior, and other treatment-relevant information. A realistic short-term goal for crisis intervention is agreed upon (e.g., enhancing sleep, managing depressive symptoms, reducing suicide ideation). Additional treatments mentioned above are prescribed individually. The multiprofessional team meets daily from Monday to Friday to discuss suicide risk, evaluate and modify crisis intervention steps or goals, manage transference and countertransference issues, evaluate and modify the diagnosis, and plan for discharge.
Exclusion criteria for the present study were lack of language skills because of migration from non-German speaking countries, discharge or transfer to another unit before completing the assessment, dementia, and acute psychotic symptoms. Two assessments were scheduled. The intake assessment was completed after the intake interview with the responsible psychiatrist and psychologist/psychotherapist. For this assessment, eligible patients filled out a set of questionnaires online, if possible within the first two days of their stay. The questionnaires assessed suicide-related risk- and protective factors, sociodemographic variables, and treatment expectation (see below for the instruments used for this study and  for other instruments not used in this study). In case of acute psychosis or severe depression, patients completed the questionnaires as soon as their condition improved. The second assessment was scheduled within two days before discharge and included only a subset of questionnaires on suicide ideation, depression, hopelessness, and working alliance. If the length of stay was too short, no additional assessment was made before discharge.
Patients were informed that data were stored safely (on a separate server only accessed by selected researchers and without identifying names except the protected clinic code) and that the data could only be used by the research team and the responsible clinician. The responsible clinicians were only informed about certain outcome variables (suicide ideation, hopelessness, depression, treatment expectancy, working alliance) but not about sexual orientation variables. All study participants gave written consent. The study was approved by the ethics commission of Salzburg.
Following recommendations , the intake assessment included three dimensions of sexual orientation. Sexual attraction was assessed with the item “To whom are you sexually attracted?” (men, women, men and women), with the possible response options in brackets. Responses were categorized in heterosexual, bisexual, and same-sex attraction, and the latter categories were subsumed under the SM attraction category.
Sexual behavior was solicited with two items: “With how many men did you ever have sex with?” (with none, one man, two men, three or more men) and a corresponding item for sex with women. Responses were categorized into heterosexual (exclusively other-sex sexual behavior), bisexual (sex with both men and women), same-sex (exclusively same-sex behavior), and a separate “no sex” category. Homosexually or bisexually active participants were collapsed into the SM behavior category.
Sexual identity was assessed with “What describes your sexual orientation best?” [heterosexual (sexually interested in the other sex), predominantly heterosexual, bisexual (sexually interested in men as well as women), gay/lesbian/homosexual (sexually interested in the same sex), I am not sure, I don’t understand this question, other]. In contrast to the items on behavior and attraction, participants also had an open response field after the “other” option. We decided to include “homosexual” as a response option, since in Austria, some still prefer this term to “schwul” (the German expression for “gay”).
Since subgroups of sexual orientation were too small to allow statistical analysis with adequate power, we created a binary sexual orientation variable. Patients were classified as SM if they had any indicator of SM in the identity, behavior, or attraction variable. All others were classified as heterosexual. This binary variable was used for all analysis. Results for each of the three dimensions of sexual orientation may be obtained upon request.
The three relevant variables were assessed at intake and before discharge. Suicide ideation was measured with Beck’s Scale for Suicide Ideation  (Cronbach’s alpha r α = .94 for Items 1 to 19); suicide attempt status (yes/no) was derived from the related item of that scale. Hopelessness was assessed with Beck’s Hopelessness-Scale  (r α = .88), depression with Beck’s Depression Inventory, Version II  (r α = .91). These three instruments are typically used to assess treatment outcome in related studies [52,53,54,55], since they are established measures of suicide risk [56, 57] with sound psychometric properties [57, 58] and therefore recommended in suicide prevention research . The calculation of the outcome measures (intake-discharge quantitative differences and treatment responder analysis) is described below.
Treatment expectancy and working alliance
Treatment expectancy was included in the intake assessment with the item “How much do you trust that the treatment at this department will help you solve your problems?” Participants had to respond on a visual-analogue scale ranging from 0 “no trust” to 100 “maximum trust.” A similar procedure had been used previously . Therapeutic alliance was included in the discharge assessment, using the German translation of the Working Alliance Inventory  (r α = .94). Examples of items are “I believe my therapist likes me,” or “My therapist and I collaborate on setting goals for my therapy.” Four patients had more than two missing items in this questionnaire; following the manual , we did not calculate a score for these patients. We replaced one or two missing value(s) with the median of the completed items in n = 19 patients.
Patients were first diagnosed according to the ICD-10 in the initial intake interview with a psychiatrist and a psychologist or psychotherapist and by studying the medical records or the diagnosis given by referring institutions. Each patient then had daily contact with a psychiatrist, a clinical psychologist or psychotherapist, and a nurse. In the daily meetings of the multiprofessional team, the appropriateness of the diagnosis was discussed and, if necessary, adjusted. The diagnosis at discharge was used as a control variable for the present study. Only the main categories of the ICD-10 (F0 – F9) were used for the analysis (see Table 1).
Single items were used to assess gender, income, nationality, and mother language. Age was calculated from the date of birth. Degree of education was coded ordinally as compulsory schooling (1), compulsory schooling plus apprenticeship (2), A-level (3), and academic degree (applied sciences: 4, regular university: 5). Since 72 participants replied with “other,” this could not be categorized ordinally. They were coded as missing and then imputed with R’s “transcan” function  by using all other sociodemographic variables for prediction. Similarly, since the item on income was not mandatory, missing data (n = 91, 14%) were imputed.
Length of stay
This was calculated from the date of admission and discharge from the CI-SP department. In some cases, this also included days of temporary transfers to other departments, most often the closed ward. Since length of stay was skewed (a few patients had very long stays, range 3–422), we log-transformed the variable in the multivariate analysis.
We used two measures for treatment outcome. First, quantitative change was measured by calculating the differences between intake and discharge levels of suicide ideation, hopelessness, and depression. Sexual orientation differences (SM vs. heterosexual patients) were then calculated with t-tests and with multivariate linear regression models to adjust for potential confounders. Second, we used a treatment responder analysis . To be classified as a responder, the intake level had to be above the clinical cut-off with an improvement of at least 50% of the clinical range. Furthermore, the overall improvement had to be at least 25% in the general range of the scale. This is important because if a patient’s initial level is only slightly above the cut-off, then it is too easy to gain >50% possible change in the clinical range. Nonresponders were those who did not achieve the improvement just described. Those who were not in the pathological range in the intake assessment were excluded from the responder analysis. Sexual-orientation differences in responder status were based on simple odds ratios or on multivariate logistic regression models to adjust for potential confounders.
Potential confounding variables in this study were sociodemographics (gender, age, level of education, income, nationality, mother language), length of stay, and diagnosis (F0-F9 codes). Since there are 18 variables, we first ran stepwise regression analysis with backwards elimination with only the sociodemographics and with only the 10 diagnoses to identify important predictors for the dependent variable in question. We also inspected interactions of each confounding variable with sexual orientation (detailed results in supplement). Significant interactions are reported and also included in the full model. Regression results for the full models are reported in detail in Additional file 1.
For a priori power analysis, we assumed a prevalence of 15% SM individuals in our sample. To detect small effects (d = 0.3) with ɑ = .05 and 80% power for comparing means of two independent samples (SM vs. heterosexuals) with one-sided t-tests, a sample size of 529 heterosexuals and 79 SM individuals would be needed (671 and 101 for two - tailed tests, respectively). Our sample included 502 heterosexuals and 131 SM patients, thus a post hoc analysis revealed an actual power of 92% in one-sided and 86% in two-sided t-tests .
Missing data were impossible for most variables, since each item had to be completed in order to move on to the next item. Due to a programming error, however, it was possible to skip items or to fill in impossible values only for a few variables at the very beginning of the assessment period and only for a few variables as this was corrected early. Details on imputation are given above for the corresponding instruments.
We calculated Cohen’s d as a measure of effect for continuous variables size by dividing the differences between the measurements of women and men by the pooled standard deviation . For categorical data, odds ratios (OR) were used to quantify effect sizes [66, 67]. To determine the associations between variables we used Spearman rank correlation. We used SPSS Version 21  and R 3.1.3  for statistical analysis.
Out of a total of 997 patients admitted during the assessment period, 834 (84%) fulfilled the inclusion criteria for the intake assessment, and 633 (63%) completed both intake and discharge assessments, comprising the analyzed sample.
With respect to sexual orientation, 12% reported bisexual or same-sex behavior and 10% reported same-sex or bisexual attraction (Table 1). For the identity item, nearly all patients who chose the “I don’t understand” or “other” response either reported exclusive heterosexual behavior/attraction or sometimes reported a related qualitative statement (e.g., “I am normal”). The few patients who responded with a SM identity (genderqueer, open to anybody, etc.) on the open-ended item also had SM behavior/attraction responses. After clarifying these “other” and “don’t understand” categories, 14% of patients were classified as having a SM identity. Overall, 21% reported at least one indicator of SM status.
Differences by sexual orientation
SM patients were significantly younger (d = 0.36) and more likely female (OR = 1.59, 95%-CI 1.08-2.38), compared to heterosexual patients, but the differences were small. No other sexual orientation differences achieved statistical significance (Table 2).
Length of stay
SM and heterosexual patients did not differ significantly in their length of stay (d = 0.22).
The distribution of diagnosis only differed significantly for personality disorders, where SM patients were almost twice as likely to receive such a diagnosis, compared to heterosexual patients (Table 2).
At both intake and discharge assessment, there were only small sexual orientation differences (d < 0.27) in suicide ideation, hopelessness, and depression. Only suicide ideation achieved statistical significance, indicating that SM patients had slightly higher levels than heterosexual patients at intake and discharge (Table 2).
Treatment outcome (primary study aim)
There was a close to zero difference in the improvement of suicide ideation between heterosexual and SM patients in the unadjusted (d = 0.06) and adjusted analysis (Table 3). Only F7 diagnosis (mental retardation) interacted significantly with sexual orientation: SM patients with F7 diagnosis showed a large improvement in suicide ideation compared to heterosexual patients with F7 diagnosis (d = 1.19), whereas no sexual orientation difference occurred among patient without F7 diagnosis (d = 0.02).
The odds for not responding to treatment were comparable between heterosexual and SM patients in the unadjusted analysis (OR = 1.20, 95%-CI 0.69–2.04). When adjusting for confounders, SM patients were somewhat less likely nonresponders, compared to heterosexuals (OR = 0.44, 95%-CI 0.18–1.05), but the difference did not reach statistical significance (p = .10). Significant interactions occurred for F3 diagnosis (affective disorders) and sexual orientation: Among patients without F3 diagnosis, SM patients were more likely nonresponders than heterosexual patients (OR = 3.56), whereas only a small sexual orientation difference was found among those with a F3 diagnosis (OR = 0.81). There was a reversed interaction effect for F6 diagnosis (personality disorders): Among patients without F6 diagnosis, SM patients were less likely nonresponders than heterosexual patients (OR = 0.64), whereas the reverse association was found among those with a F6 diagnosis (OR = 2.65). Finally, length of stay interacted with sexual orientation: Among heterosexual patients, length of stay did not differ between responders and nonresponders (d = 0.04); among SM patients nonresponders had a longer stay than responders (d = 0.69).
There was a close to zero difference in improvement of hopelessness between heterosexual and SM patients in the unadjusted (d = 0.03) and adjusted analysis (Table 3). Only F6 diagnosis (personality disorders) interacted significantly with sexual orientation: among heterosexual patients there was no difference in change of hopelessness between those with and without F6 diagnosis, whereas SM patients without F6 diagnosis showed slightly more improvement (d = 0.14) and SM patients without F6 diagnosis slightly less improvement (d = 0.26) than heterosexual patients.
The odds for not responding to treatment were comparable between heterosexual and SM patients in the unadjusted analysis (OR = 1.25, 95%-CI 0.75–2.15). In the adjusted analysis, SM patients were somewhat more likely to be nonresponders compared to heterosexuals (OR = 2.01, 95%-CI 0.97–4.16) but the difference did not reach statistical significance (p = .06). Significant interactions with sexual orientation only occurred for F4 diagnosis (anxiety, dissociative, stress-related, somatoform disorders): among those without an F4 diagnosis, SM patients were more likely to be nonresponders than heterosexual patients (OR = 2.12), whereas among those with an F4 diagnosis the reverse association was found (OR = 0.51).
There was a close to zero difference in improvement in depression between heterosexual and SM patients in the unadjusted model (d = 0.02). After adjusting for confounders, SM patients showed significantly more improvement than heterosexual patients (Table 3). Only length of stay interacted significantly with sexual orientation: for patients with a stay of maximum 14 days, SM patients had slightly greater improvement in depression than heterosexuals (d = 0.28), whereas among those with longer stays (> 30 days), SM patients demonstrated somewhat lesser improvement (d = 0.34).
The odds for responding to treatment were comparable between heterosexual and SM patients in the unadjusted analysis (OR = 1.06, 95%-CI 0.67–1.63). In the adjusted analysis, SM patients were significantly less likely to be nonresponders, compared to heterosexuals (OR = 0.77, 95%-CI 0.46–1.28). Significant interactions with sexual orientation only occurred for length of stay: there was no sexual orientation difference among responders (d = 0.06), whereas among nonresponders, SM patients had longer stays than heterosexual patients (d = 0.73).
Treatment expectancy and working alliance (secondary study aim)
There was a close to zero difference (d = 0.02) between heterosexual and SM patients in the unadjusted analysis (Table 3). However, in the adjusted model, sexual orientation became a significant predictor (p = .046), meaning that SM patients had more positive treatment expectations than heterosexual patients. Only nationality interacted with sexual orientation: there were no sexual orientation differences in expectation among Austrians (d = 0.13), whereas SM non-Austrians scored somewhat lower in expectancy than heterosexual non-Austrians (d = 0.49).
There was a nonsignificant and very small (d = 0.09) difference between heterosexual and SM patients in the unadjusted analysis (Table 3). In the adjusted model, sexual orientation became a significant predictor (p = .02), meaning that SM patients had higher levels of working alliance compared to heterosexual patients. Only nationality and length of stay interacted with sexual orientation: There were only slight sexual orientation differences in working alliance among Austrians (d = 0.13), whereas SM non-Austrians scored much lower in working alliance levels than heterosexual non-Austrians (d = 0.82). With respect to treatment length, sexual orientation differences were small for short-term stays and disappeared for longer stays.
In our study, we compared SM and heterosexual patients at risk for suicide in a public psychiatric setting. The primary goal of our study was to compare treatment outcome by sexual orientation, and the secondary aim to compare treatment expectancy and working alliance by sexual orientation.
With respect to treatment outcome, contrary to our expectations, there were no notable differences by sexual orientation in all unadjusted analyses for change in suicide ideation, hopelessness and depression. No significant differences were found in the related responder analysis, either. The observed differences lacked statistical significance and were of small magnitude. When adjusting for potential confounders (sociodemographics, diagnosis, and length of stay), the sexual orientation differences became significant for depression, but contrary to the hypothesis: SM patients demonstrated larger improvements in levels of depression and a higher likelihood of responding.
Concerning our secondary study aim, we found that SM and heterosexual patients were comparable in their levels of treatment expectancy and working alliance. When adjusting for confounders, SM patients demonstrated significantly greater treatment expectation and better working alliance than heterosexual patients.
The results are surprising, given that previous studies reported barriers to healthcare for SM patients leading potentially to poorer treatment outcomes. There are several possible explanations for these findings, some of them related to potential limitations of our study.
It remains unknown if the CI-SP department is representative of other mental health departments with regard to SM-specific competencies and SM-affirmative attitudes among the staff. For example, an English study found that, overall, satisfaction with healthcare was lower for SM than heterosexual patients, but there was great variation between general practices, with some practices evaluated similarly by SM and heterosexual patients . In the CI-SP department, the staff has not received any formal training in SM-specific competencies. However, some of the authors of this paper have published on suicide risk of SM individuals and are also members of the therapeutic team, with one of them being openly gay. This could have led to changes in attitudes and increased awareness about SM issues , thereby improving care for SM patients. Replication studies in other clinical settings are needed. Staff attitudes and expertise were not assessed in this study, but should be assessed in future studies.
An alternative explanation is related to the contextual model of psychotherapeutic change . According to this model, treatment success results from contextual factors, including empathic and real therapeutic relationship, creation of positive expectations, and a shared working model of the problem. In contrast, specific ingredients, such as specific forms of therapies or adherence to protocols are of minor importance for treatment success. If SM-specific competencies constitute specific ingredients in therapy, then our results are not surprising. This alternative explanation is also supported by the finding that SM and heterosexual patients do not differ on two important common factors of treatment (working alliance and treatment expectation). Perhaps the lack of SM-specific competencies can be overcome by a strong therapeutic alliance, in the sense of “I had to educate my therapist about the realities of being a lesbian woman, but he/she was a kind person and really tried to help me.” In line with this argument, although a large majority of gay men (including those with major depression) reported that they would more likely present and speak openly in a gay-friendly treatment setting, only a plurality felt that gay-friendly providers would improve the treatment outcome . However, there may also be a tradeoff at some point whereby the lack in SM competencies starts to have a negative impact on treatment success. A strong therapeutic alliance can mitigate disclosure of sexual orientation, but recognizing the heteronormative value system by the clinician is a key to establish a strong therapeutic relationship . Furthermore, in psychotherapy, therapists who were perceived as helpful clearly had more SM-specific competencies than unhelpful therapists . Future studies could use a similar methodology as Liddle’s  to study psychiatric settings.
The argument above could also be applied to the interesting finding that treatment expectation was comparable between SM and heterosexual patients. If SM patients had negative experiences in healthcare or in society in general, then they should have lower treatment expectancies than heterosexuals before the beginning of treatment. However, we assessed treatment expectancy after the intake interview, which is done in a patient-centered, narrative style known to enhance therapeutic alliance and outcome with suicidal patients [38, 71]. Thus, the intake interview itself could have improved treatment expectancy for SM patients, but this remains an issue for further empirical investigations.
Our findings are relevant both in terms of sexual orientation-based health disparities and mental health service provision. A wealth of literature [3, 16] outlines the myriad ways SM status is a risk factor for poor mental health and elevated suicide risk. While many approaches to health disparity reduction focus on specific cultural competence training (see  for review), our data suggest the potential for a common factors approach to reduce such disparities in an inpatient paradigm. Effective contextual factors are more easily and generally applicable to treatment settings, offering promise for further implementation and study.
Our results could lack validity through biased self-reports, a problem that is shared by all research on hidden populations. For example, some SM patients may have remained undisclosed in the assessment – i.e., identified as heterosexual or have chosen the “I don’t understand” option – or some may have refused to take part or complete the assessment. We did not use open-ended response options in the items on sexual behavior and attraction, and this may have been problematic for some people, for example those with lacking sexual attraction. The percentage of patients who identified as SM resembles results from the general US population  and was slightly lower than in a study of patients presenting at an emergency department , suggesting that underreporting may not be severe. However, given that SM individuals are at higher risk for suicide and mental disorders, they should be overrepresented among patients, but only if they use mental health services at least as frequently as heterosexuals. Nonetheless, underreporting is a reality, proven by a few patients who disclosed during treatment but refused to do so in the assessment. Coming out during treatment could interact with treatment outcome but we did not assess disclosure to the clinicians systematically. Furthermore, problems with coming out is a frequent cause of suicide attempts among gay men ; thus, underreporting and suicide risk are likely correlated among SM patients. Unfortunately, we could not systematically assess such misclassified individuals. This bias would be especially problematic if misclassification varies with the outcome variables – i.e., if those with low treatment expectancy or treatment success hide their SM status. Perhaps some of the bias can be reduced with follow-up studies whereby patients are assessed anonymously by a third party. Despite these limitations, we found the assessment of sexual orientation feasible, in line with a recent US study , and encourage measures of sexual orientation in assessment instruments, as recommended by the Institute of Medicine  in order to improve care for SM patients [76, 77].
Another limitation is that we did not assess transgender and intersex but only a dichotomized category of gender (male/female) which could again produce biases. Moreover, this is itself an example of lacking minority-specific competency, in this case about gender minorities which should be avoided. This has been corrected this in the meantime. We also did not have information of the type of medication for individual patients available for analysis. However, this would only be important if medication varies with sexual orientation or if medication works differently for SM than heterosexual patients, which is unlikely. Of course, medication varies with diagnoses and some diagnoses were disproportionately represented among SM patients. This is why we investigated all interaction effects with diagnosis and adjusted for diagnosis in the multivariate analysis.
The sample size limits the precise estimation for subgroups of SM individuals as well as subgroups based on gender, ethnic background, diagnosis, and other relevant factors. Additional analysis for the three dimensions of sexual orientation (provided upon request) suggest that the results for subgroups of SM individuals did not differ significantly, with a few exceptions that are contrary to our hypothesis. Bisexually behaving patients had higher levels of treatment expectation, same-sex attracted patients had higher levels of working alliance, other identified patients (most of them heterosexually classified) had less improvement in depression. These results could be false positives resulting from multiple testing, since nearly all differences are far from being significant in Tukey post hoc comparison tests (results not shown). On the other hand, given the small subsamples, even larger differences will not become statistically significant. The same applies to the interaction effects, where – out of the many gender or diagnosis interaction effects with sexual orientation – only a few were statistically significant. Of note, it seems that SM patients of non-Austrian nationality responded less optimally to treatment, and they also had lower treatment expectancy and working alliance, compared to SM Austrians. This finding deserves further exploration, since these results may be false positives, given the large number of interactions. Larger replication studies are needed to allow subgroup analysis and examine interaction effects.
We did not record if patients had had a history of prior hospitalization at the CI-SP department. If patients had been in the CI-SP department before, their previous experience with the treating health professionals might have influenced their current treatment expectation.
Despite these limitations, this study is, to our knowledge, the first to investigate treatment success and related variables in relation to sexual orientation in a sample of patients at risk for suicide in a general psychiatric hospital and thus has more ecological validity than previous studies.
SM and heterosexual patients at risk for suicide in a public psychiatric hospital setting did not differ significantly in treatment outcome, treatment expectation, and working alliance. This is contrary to what is expected from the literature where barriers for SM patients are reported which have a negative impact on treatment outcome and satisfaction with treatment. Perhaps SM-specific competencies in our clinical setting were not representative (this was not assessed, unfortunately), or common factors of therapy can, to a certain degree, compensate for a lack of SM-specific competencies. Replications in other treatment settings are needed before these findings can be generalized.
- 95%-CI :
95-Percent confidence interval
Crisis intervention and suicide prevention
King M, Semlyen J, Tai SS, Killaspy H, Osborn D, Popelyuk D, Nazareth I. A systematic review of mental disorder, suicide, and deliberate self harm in lesbian, gay and bisexual people. BMC Psychiatry. 2008;8(1):70.
Plöderl M, Tremblay P. Mental health of sexual minorities. A systematic review. Int Rev Psychiatry. 2015;27(5):367–85.
Haas AP, Eliason M, Mays VM, Mathy RM, Cochran SD, D'Augelli AR, Silverman MM, Fisher PW, Hughes T, Rosario M, et al. Suicide and suicide risk in lesbian, gay, bisexual, and transgender populations: review and recommendations. J Homosex. 2011;58(1):10–51.
Harris EC, Barraclough B. Suicide as an outcome for mental disorders. A meta-analysis. Br J Psychiatry. 1997;170:205–28.
Holmstrand C, Bogren M, Mattisson C, Bradvik L. Long-term suicide risk in no, one or more mental disorders: the Lundby study 1947-1997. Acta Psychiatr Scand. 2015;132(6):459–69.
Meyer IH. Prejudice, social stress, and mental health in lesbian, gay, and bisexual populations: conceptual issues and research evidence. Psychol Bull. 2003;129(5):674–97.
Hatzenbuehler ML. How does sexual minority stigma "get under the skin"? A psychological mediation framework. Psychol Bull. 2009;135(5):707–30.
Pachankis JE. A transdiagnostic minority stress treatment approach for gay and bisexual men's syndemic health conditions. Arch Sex Behav. 2015;44(7):1843–60.
Stall R, Friedman M, Catania JA. Interacting epidemics and gay men’s health: a theory of syndemic production among urban gay men. In: Wolitski RJ, Stall R, Valdiserri RO, editors. Unequal opportunity: health disparities affecting gay and bisexual men in the United States. New York: Oxford University Press; 2008. p. 251–74.
Plöderl M, Sellmeier M, Fartacek C, Pichler EM, Fartacek R, Kralovec K. Explaining the suicide risk of sexual minority individuals by contrasting the minority stress model with suicide models. Arch Sex Behav. 2014;43(8):1559–70.
Burks AC, Cramer RJ, Henderson CE, Stroud CH, Crosby JW, Graham J. Frequency, nature, and correlates of hate crime victimization experiences in an urban sample of lesbian, gay, and bisexual community members. J Intern Viol. 2015;
Burton CM, Marshal MP, Chisolm DJ, Sucato GS, Friedman MS. Sexual minority-related victimization as a mediator of mental health disparities in sexual minority youth: a longitudinal analysis. J Youth Adolesc. 2013;42(3):394–402.
Kuyper L, Fokkema T. Minority stress and mental health among Dutch LGBs: examination of differences between sex and sexual orientation. J Couns Psychol. 2011;58(2):222–33.
Shilo G, Mor Z. The impact of minority stressors on the mental and physical health of lesbian, gay, and bisexual youths and young adults. Health Soc Work. 2014;39(3):161–71.
Ferlatte O, Dulai J, Hottes TS, Trussler T, Marchand R. Suicide related ideation and behavior among Canadian gay and bisexual men: a syndemic analysis. BMC Public Health. 2015;15
Institute of Medicine. The health of lesbian, gay, bisexual, and transgender people: building a foundation for better understanding. Washington, DC: The National Academic Press; 2011.
Herek GM, Gillis JR, Cogan JC. Internalized stigma among sexual minority adults: insights from a social psychological perspective. J Couns Psychol. 2009;56(1):32–43.
Cant B. Exploring the implications for health professionals of men coming out as gay in healthcare settings. Health Soc Care Comm. 2006;14(1):9–16.
Eliason MJ, Schope R. Does “don't ask don't tell” apply to health care? Lesbian, gay, and bisexual people's disclosure to health care providers. J Gay Lesbian Med Assoc. 2001;5(4):125–34.
Eliason MJ, Dibble SL, Robertson PA. Lesbian, gay, bisexual, and transgender (LGBT) physicians' experiences in the workplace. J Homosex. 2011;58(10):1355–71.
Bartlett A, Smith G, King M. The response of mental health professionals to clients seeking help to change or redirect same-sex sexual orientation. BMC Psychiatry. 2009;9:11.
Legal L. When health care isn’t caring: Lambda legal’s survey of discrimination against LGBT people and people with HIV. New York: Lambda Legal; 2010.
Willging CE, Salvador M, Kano M. Unequal treatment: mental health care for sexual and gender minority groups in a rural state. Psychiatr Serv. 2006;57(6):867–70.
Dean MA, Victor E, Grimes LG. Inhospitable healthcare spaces: why diversity training on LGBTQIA issues is not enough. Journal of bioethical inquiry. 2016;
Hinchliff S, Gott M, Galena E. I daresay I might find it embarrassing': general practitioners' perspectives on discussing sexual health issues with lesbian and gay patients. Health Soc Care Comm. 2005;13(4):345–53.
Parameshwaran V, Cockbain BC, Hillyard M, Price JR. Is the lack of specific lesbian, gay, bisexual, transgender and queer/questioning (LGBTQ) health care education in medical school a cause for concern? Evidence from a survey of knowledge and practice among UK medical students. J Homosex. 2017;64(3):367–81.
Khan A, Plummer D, Hussain R, Minichiello V. Does physician bias affect the quality of care they deliver? Evidence in the care of sexually transmitted infections. Sex Transm Infect. 2008;84(2):150–1.
East JA, El Rayess F. Pediatricians' approach to the health care of lesbian, gay, and bisexual youth. J Adolesc Health. 1998;23(4):191–3.
Elliott MN, Kanouse DE, Burkhart Q, Abel GA, Lyratzopoulos G, Beckett MK, Schuster MA, Roland M. Sexual minorities in England have poorer health and worse health care experiences: a national survey. J Gen Intern Med. 2015;30(1):9–16.
Bernstein KT, Liu KL, Begier EM, Koblin B, Karpati A, Murrill C. Same-sex attraction disclosure to health care providers among New York City men who have sex with men: implications for HIV testing approaches. Arch Intern Med. 2008;168(13):1458–64.
Law M, Mathai A, Veinot P, Webster F, Mylopoulos M. Exploring lesbian, gay, bisexual, and queer (LGBQ) people's experiences with disclosure of sexual identity to primary care physicians: a qualitative study. BMC Fam Pract. 2015;16
Willging CE, Salvador M, Kano M. Pragmatic help seeking: how sexual and gender minority groups access mental health care in a rural state. Psychiatr Serv. 2006;57(6):871–4.
Wang J, Hausermann M, Weiss MG. Mental health literacy and the experience of depression in a community sample of gay men. J Affect Disord. 2014;155:200–7.
Liddle BJ. Therapist sexual orientation, gender, and counseling practices as they relate to ratings of helpfulness by gay and lesbian clients. J Couns Psychol. 1996;43(4):394–401.
Meyer IH, Teylan M, Schwartz S. The role of help-seeking in preventing suicide attempts among lesbians, gay men, and bisexuals. Suicide Life Threat Behav. 2015;45(1):25–36.
Wampold BE, Imel ZE. The great psychotherapy debate: the evidence for what makes psychotherapy work. New York, NY: Routledge; 2015.
Gysin-Maillart A, Schwab S, Soravia L, Megert M, Michel K. A novel brief therapy for patients who attempt suicide: a 24-months follow-up randomized controlled study of the attempted suicide short intervention program (ASSIP). PLoS Med. 2016;13(3):e1001968.
Michel K, Jobes DA. Building a therapeutic alliance with the suicidal patient. Washington, DC: American Psychological Association; 2011.
Ilgen MA, Czyz EK, Welsh DE, Zeber JE, Bauer MS, Kilbourne AM. A collaborative therapeutic relationship and risk of suicidal ideation in patients with bipolar disorder. J Affect Disord. 2009;115(1–2):246–51.
Bedics JD, Atkins DC, Comtois KA, Linehan MM. Treatment differences in the therapeutic relationship and introject during a 2-year randomized controlled trial of dialectical behavior therapy versus nonbehavioral psychotherapy experts for borderline personality disorder. J Consult Clin Psych. 2012;80(1):66–77.
Benedetti F. Placebo effects. USA: Oxford University Press; 2014.
Cruz M, Pincus HA. Research on the influence that communication in psychiatric encounters has on treatment. Psychiatr Serv. 2002;53(10):1253–65.
Rudd MD, Berman AL, Joiner TE Jr, Nock MK, Silverman MM, Mandrusiak M, Van Orden K, Witte T. Warning signs for suicide: theory, research, and clinical applications. Suicide Life Threat Behav. 2006;36(3):255–62.
Mann JJ, Apter A, Bertolote J, Beautrais A, Currier D, Haas A, Hegerl U, Lonnqvist J, Malone K, Marusic A, et al. Suicide prevention strategies: a systematic review. JAMA. 2005;294(16):2064–74.
Granello D. A suicide crisis intervention model with 25 practical strategies for implementation. J Ment Health Couns. 2010;32(3):218–35.
Dinhobl R, Aistleitner U, Plöderl M, Fartacek C. Dialektische Verhaltenstherapieelemente in der existenzanalytisch geprägten stationären Krisenintervention mit suizidalen Borderline-Patienten. Suizidprophylaxe. 143:155–60.
Kralovec K, Kunrath S, Fartacek C, Pichler EM, Plöderl M. The gender-specific associations between religion/spirituality and suicide risk in a sample of Austrian psychiatric inpatients. Suicide Life Threat Behav. 2017; doi:10.1111/sltb.12349.
Saewyc EM, Bauer GR, Skay CL, Bearinger LH, Resnick MD, Reis E, Murphy A. Measuring sexual orientation in adolescent health surveys: evaluation of eight school-based surveys. J Adolesc Health. 2004;35(4):345. e341-315
Beck AT, Steer RA. Beck scale for suicide ideation (BSS). San Antonio, TX: The Psychological Corporation; 1991.
Krampen G. Skalen zur Erfassung von Hoffnungslosigkeit (H-Skalen). Deutsche Bearbeitung und Weiterentwicklung von Aaron T. Beck. Handanweisung. Göttingen, Germany: Hogrefe; 1994.
Kuhner C, Burger C, Keller F, Hautzinger M. Reliability and validity of the revised Beck depression Inventory (BDI-II). Results from German samples. Nervenarzt. 2007;78(6):651–6.
Brown GK, Ten Have T, Henriques GR, Xie SX, Hollander JE, Beck AT. Cognitive therapy for the prevention of suicide attempts: a randomized controlled trial. JAMA. 2005;294(5):563–70.
Rudd MD, Bryan CJ, Wertenberger EG, Peterson AL, Young-McCaughan S, Mintz J, Williams SR, Arne KA, Breitbach J, Delano K, et al. Brief cognitive-behavioral therapy effects on post-treatment suicide attempts in a military sample: results of a randomized clinical trial with 2-year follow-up. Am J Psychiatry. 2015;172(5):441–9.
Bryan CJ, Mintz J, Clemans TA, Leeson B, Burch TS, Williams SR, Maney E, Rudd MD. Effect of crisis response planning vs. contracts for safety on suicide risk in U.S. Army soldiers: a randomized clinical trial. J Affect Disord. 2017;212:64–72.
Bryan CJ, Clemans TA, Hernandez AM, Mintz J, Peterson AL, Yarvis JS, Resick PA, Consortium SS. Evaluating potential iatrogenic suicide risk in trauma-focused group cognitive behavioral therapy for the treatment of ptsd in active duty military personnel. Depress Anxiety. 2016;33(6):549–57.
Mann JJ, Waternaux C, Haas GL, Malone KM. Toward a clinical model of suicidal behavior in psychiatric patients. Am J Psychiatry. 1999;156(2):181–9.
Brown GK. A review of suicide assessment measures for intervention research with adults and older adults. National Institute of Mental Health. 2001. http://www.sprc.org/sites/default/files/migrate/library/BrownReviewAssessmentMeasuresAdultsOlderAdults.pdf. Accessed 15 April 2017.
Kreuze E, Lamis DA. A review of psychometrically tested instruments assessing suicide risk in adults. Omega (Westport). 2017;30222816688151
National Guideline Clearinghouse. Guideline summary: Clinical practice guideline for the prevention and treatment of suicidal behaviour. Agency for Healthcare Research and Quality. 2012. https://www.guideline.gov/summaries/summary/48046/clinical-practice-guideline-for-the-prevention-and-treatment-of-suicidal-behaviour. Accessed 2 May 2017.
Vogel PA, Hansen B, Stiles TC, Gotestam KG. Treatment motivation, treatment expectancy, and helping alliance as predictors of outcome in cognitive behavioral treatment of OCD. J Behav Ther Exp Psychiatry. 2006;37(3):247–55.
Wilmers F, Munder T, Leonhart R, Herzog T, Plassmann R, Barth J, Linster HW. Die deutschsprachige version des working alliance Inventory–short revised (WAI-SR). Ein schulenübergreifendes, ökonomisches und empirisch validiertes instrument zur Erfassung der therapeutischen Allianz [the German version of the working alliance Inventory - short revised (WAI-SR) - a pantheoretic, economic, and empirically validated instrument to assess the therapeutic alliance]. Klinische Diagnostik Evaluat. 2008;1(3):343–58.
Harrell F. E. Jr., Charles Dupont, and many others: Hmisc: Harrell Miscellaneous. In.; 2016.
Hiller W, Schindler AC, Lambert MJ. Defining response and remission in psychotherapy research: a comparison of the RCI and the method of percent improvement. Psychother Res. 2012;22(1):1–11.
Faul F, Erdfelder E, Lang AG, Buchner A. G*power 3: a flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behav Res Methods. 2007;39(2):175–91.
Rosenthal R, Rosnow RL. Essentials of behavioral research: methods and data analysis. 2nd ed. New York: McGraw Hill; 1991.
Breaugh JA. Effect size estimation: factors to consider and mistakes to avoid. J Manage. 2003;29(1):79–97.
Haddock CK, Rindskopf D, Shadish WR. Using odds ratios as effect sizes for meta-analysis of dichotomous data: a primer on methods and issues. Psychol Methods. 1998;3(3):339–53.
Corp IBM. IBM SPSS statistics for windows, version 21.0. IBM Corp: Armonk, NY; 2012.
R Core Team. R: a language and environment for statistical computing: Vienna, Austria, R Foundation for Statistical Computing; 2016.
Burke SE, Dovidio JF, Przedworski JM, Hardeman RR, Perry SP, Phelan SM, Nelson DB, Burgess DJ, Yeazel MW, van Ryn M. Do contact and empathy mitigate bias against gay and lesbian people among heterosexual first-year medical students? A report from the medical student CHANGE study. Acad Med. 2015;90(5):645–51.
Michel K, Dey P, Stadler K, Valach L. Therapist sensitivity towards emotional life-career issues and the working alliance with suicide attempters. Arch Suicide Res. 2004;8(3):203–13.
Purnell L. Are we really measuring cultural competence? Nurs Sci Q. 2016;29(2):124–7.
Chandra A, Mosher WD, Copen C, Sionean C. Sexual behavior, sexual attraction, and sexual identity in the United States: data from the 2006-2008 National Survey of family growth. Natl Health Stat Rep. 2011(36):1–36.
Currier GW, Brown G, Walsh PG, Jager-Hyman S, Chaudhury S, Stanley B. Screening for sexual orientation in psychiatric emergency departments. West J Emerg Med. 2015;16(1):80–4.
Wang J, Plöderl M, Hausermann M, Weiss MG. Understanding suicide attempts among gay men from their self-perceived causes. J Nerv Ment Dis. 2015;203(7):499–506.
Callahan EJ, Hazarian S, Yarborough M, Sanchez JP. Eliminating LGBTIQQ health disparities: the associated roles of electronic health records and institutional culture. Hast Cent Rep. 2014;44(Suppl 4):S48–52.
Callahan EJ, Sitkin N, Ton H, Eidson-Ton WS, Weckstein J, Latimore D. Introducing sexual orientation and gender identity into the electronic health record: one academic health center's experience. Acad Med. 2015;90(2):154–60.
We thank Reinhold Fartacek for his support in realizing the study, especially for implementing the assessment into the clinical routine.
No funding was involved in the data collection, analysis, interpretation of data and writing the manuscript.
Availability of data and materials
The datasets analyzed during the current study are not publicly available due to privacy protection of the patients. However, additional analyses, if reasonable, are available from the corresponding author. Additional analyses are possible when the privacy of the patients remains protected (e.g., no information about individual cases or too small subgroups). Depending on the workload associated with the additional analysis, the necessary R-code has to be provided.
Study planning and design: MP, SK, RJC, JW, LH, CF, Data collection and supervision: MP, CF, LH, Data preparation and statistical analysis: MP, SK, Manuscript preparation: MP, SK, RJC, JW, LH, CF. All authors read and approved the final manuscript.
The authors declare that they have no competing interests.
Consent for publication
Ethics approval and consent to participate
All study participants gave written consent. The study was approved by the ethics commission of Salzburg, Austria, reference number 415-E/1153/6–2014.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Plöderl, M., Kunrath, S., Cramer, R.J. et al. Sexual orientation differences in treatment expectation, alliance, and outcome among patients at risk for suicide in a public psychiatric hospital. BMC Psychiatry 17, 184 (2017). https://doi.org/10.1186/s12888-017-1337-8
- Common factors