Skip to main content
  • Study protocol
  • Open access
  • Published:

Peer support for frequent users of inpatient mental health care in Uganda: protocol of a quasi-experimental study

Abstract

Background

Reducing readmissions among frequent users of psychiatric inpatient care could result in substantial cost savings to under-resourced mental health systems. Studies from high-income countries indicate that formal peer support can be an effective intervention for the reduction of readmissions among frequent users. Although in recent years formal peer support programmes have been established in mental health services in a few low- and middle-income countries (LMICs), they have not been rigorously evaluated.

Methods

This protocol describes a quasi-experimental difference-in-differences study conducted as part of a broader evaluation of the Brain Gain II peer support programme based at Butabika National Referral Hospital in Kampala, Uganda. The primary objective is to investigate whether frequent users of psychiatric inpatient care who have access to a peer support worker (PSW+) experience a greater reduction in rehospitalisation rates and number of days spent in hospital compared to those who do not have access to a peer support worker (PSW-). Frequent users, defined as adults diagnosed with either a mental disorder or epilepsy who have had three or more inpatient stays at Butabika over the previous 24 months, are referred to Brain Gain II by hospital staff on five inpatient wards. Frequent users who normally reside in a district where peer support workers currently operate (Kampala, Jinja, Wakiso and Mukono) are eligible for formal peer support and enter the PSW+ group. Participants in the PSW+ group are expected to receive at least one inpatient visit by a trained peer support worker before hospital discharge and three to six additional visits after discharge. Frequent users from other districts enter the PSW- group and receive standard care. Participants’ admissions data are extracted from hospital records at point of referral and six months following referral.

Discussion

To the best of our knowledge, this will be the first quasi-experimental study of formal peer support in a LMIC and the first to assess change in readmissions, an outcome of particular relevance to policy-makers seeking cost-effective alternatives to institutionalised mental health care.

Peer Review reports

Background

Frequent users of psychiatric inpatient care, sometimes referred to as “revolving-door”, “high-frequency” or “heavy” users, consume a disproportionate amount of the limited resources available for mental health care [1, 2]. A systematic review of mostly high-income country (HIC) studies estimated 10–30% of users of psychiatric care consume 50–80% of service resources [3].

More recent studies from low- and middle-income countries (LMICs) observe high rates of readmission and large numbers of frequent users among psychiatric inpatient populations [1, 4,5,6,7,8,9]. In Nigeria, for example, 41.4% of psychiatric inpatients at a university teaching hospital were readmitted within five years. Among those readmitted, mean number of admissions was 2.9 [6]. Meanwhile, the average cost of a single admission to a Nigerian psychiatric hospital is $3675 USD, equivalent to the cost of 90 outpatient visits [10]. Reducing readmissions among frequent users could result in substantial cost savings to under-resourced mental health systems in LMICs.

While there is very little LMIC research investigating service user perspectives on readmission, it is generally acknowledged that readmission can be a profoundly disruptive and demoralising experience [11]. In overstretched psychiatric facilities, experiences of inpatient care may be particularly distressing. Human rights watchdogs have documented overcrowding, unsanitary conditions, abusive practices and other human rights violations at psychiatric inpatient facilities in a number of LMICs [12,13,14,15,16]. According to a survey of people with mental health conditions in LMICs, psychiatric facilities are the fourth worst setting in terms of human rights violations; prisons, by comparison, are sixth [17].

There is evidence from HICs that formal peer support can reduce readmissions [18,19,20,21]. Notably, a randomised controlled trial conducted in the United States showed that frequent users receiving formal peer support in addition to standard care had an average of 0.64 fewer readmissions and nine fewer days in hospital than those receiving standard care alone [22]. It is unknown whether similar outcomes can be expected in low-resource settings, as patterns in utilisation of inpatient care can differ substantially from those in high-income countries [23]. There have been no studies to-date on the effectiveness of formal peer support as an intervention to reduce readmissions in a LMIC setting.

Broadly defined, peer support is social emotional support that is mutually offered or provided by “peers”, people with lived experience of mental, neurological or substance use disorders [21, 24]. While peer support encompasses a wide range of different interventions, distinctions are made between formal peer support and informal peer support (or “naturally occurring” peer support, as described by Repper and Carter [2011, pp.393]) [24, 25]. Formal peer support is provided through peer-led programmes or by peers recruited to support roles in traditional mental health or social services [25]. Those offering formal peer support may refer to themselves as “peer support workers” (PSWs) [26], though peer support roles can vary greatly. (For example, the American trial described above employed “peer mentors” to deliver formal peer support [22].) Regardless of their role specifications, PSWs are generally considered to be further along on the road to recovery—able to manage their illness and pursue fulfilling lives—and thus able to leverage their personal experience of recovery to support others [18, 19, 24, 25, 27, 28].

The study described in this protocol is part of a broader evaluation of the Brain Gain II project in Uganda, one of the first LMICs to establish a formal peer support programme [29,30,31]. The aim of this study is to understand the impact of a formal peer support intervention delivered by trained PSWs on service users’ readmissions. The objective is to investigate whether frequent users of inpatient care who have access to peer support (PSW+) experience a greater reduction in rehospitalisation rates and number of days spent in hospital compared to those who do not have access to peer support (PSW-).

Methods

As this is not a randomised trial, we first developed our protocol in accordance with STROBE (Strengthening The Reporting of Observational Studies in Epidemiology) guidelines [32], and then checked it against Reeves and Gaus’ (2004) adaptation of the CONSORT (Consolidated Standards of Reporting Trials) checklist for non-randomised studies [33]. Other components of the Theory of Change-driven evaluation have been protocolised and described elsewhere [34, 35]. These include: a cross-sectional survey of recovery-related knowledge, attitudes and practices among Butabika staff; a cost analysis to estimate money saved as a result of reduction in readmissions; a multi-method process evaluation; and additional qualitative methods comprising focus groups and interviews with study participants, PSWs, Butabika staff and other key stakeholders of the Brain Gain II project.

Setting

Butabika National Referral Hospital (“Butabika”) is a tertiary psychiatric facility with approximately 430 staff and 550 beds [36], though the number of inpatients often exceeds 750 and can approach nearly 1000 [15]. Butabika is located in a largely suburban area of southeastern Kampala, but patients from across Uganda access its services. Standard adult care consists primarily of on-site psychiatric and psychological interventions, and in some cases vocational training, as prescribed by hospital staff. Users in extremely vulnerable situations may be referred to a social worker for additional assistance. After discharge, users typically return to Butabika for outpatient services, attend one of four monthly community outreach clinics (each located within a 20 km radius of the hospital), or access mental health services at district hospitals closer to their homes.

Brain Gain II is a project of the Butabika Link in Uganda—a partnership between Butabika and the East London National Health Service Foundation Trust (ELFT) in the United Kingdom [31]. Brain Gain II aims to reduce the burden on inpatient care at Butabika by promoting recovery among service users on the hospital wards and following discharge. The two main components of Brain Gain II include: (1) establishing an on-campus Recovery College at Butabika, where people with lived and/or professional experience of mental or neurological disorders co-design and co-deliver a recovery-oriented training curriculum; and (2) offering formal peer support by trained PSWs to frequent users of psychiatric inpatient care at Butabika, on five hospital wards and in local communities in four districts [37].

The Butabika Recovery College (BREC) is located in the Community Recovery Team building adjacent to the Forensic Ward. Recovery Colleges are educational (as opposed to clinical) spaces that operate similarly to unaccredited adult education colleges, though with a focus on meaningful involvement of people with lived experience in all aspects of their functioning [38]. At BREC, people with lived experience (mostly PSWs) and people with professional experience (Butabika staff) co-deliver regular teaching sessions on recovery-related topics. Most teaching sessions focus on “what helps” and “what hinders” recovery, identified through a series of Recovery Listening Events held in Uganda by the Sharing Stories Group at the start of Brain Gain II [39]. However, BREC also hosts yoga, bead-making and other skills-based teaching sessions. Most students are current inpatients, though BREC is also open to outpatients, family members and hospital staff.

The Peer Support Office sits within BREC, and acts as the coordinating centre for both Recovery College trainers and PSWs. PSWs operate on five hospital wards, including the forensic ward, acute admissions wards (male and female wards), and long-stay rehabilitation wards (male and female wards). The four districts where PSWs carry out community visits include Kampala, Jinja, Wakiso and Mukono. These are located in the Central and Eastern regions of Uganda and within approximately two hours’ drive of Butabika. Communities in these districts are typically urban or suburban, and English and Luganda are widely spoken.

Study design

In keeping with Brain Gain II’s emphasis on co-production, it was agreed that both PSWs and staff should be involved in the design and conduct of the project’s evaluation. “A theory of how and why an initiative works” that can be empirically tested (Weiss 1995, p. 86 cited DeSilva et al. 2014, n.page), theory of change is increasingly recognised as a useful tool for involving diverse stakheholder groups in evaluation design [40,41,42]. Two days of Theory of Change workshops were carried out at Butabika, convening PSWs, Butabika staff and representatives of ELFT. Through guided discussions facilitated by the first author, short-, medium- and long-term outcomes were backward-mapped onto a “pathway of change”, which was further refined in consultation with the project leads at Butabika and ELFT [see Additional file 1]. Indicators were assigned to each outcome and methods proposed to measure each indicator, in order to build up the evaluation design. Methodological decisions were made with a focus on feasibility, understanding that PSWs and staff would be responsible for much of the data collection. As this was funded as a project evaluation, with limited budget for research, there was little scope for hiring external data collectors with the time and specialist expertise required to administer complex measurement tools.

For the evaluation of user-level outcomes of peer support, it was not considered appropriate by stakeholders to adopt an experimental design, in which frequent users in extremely vulnerable situations who could otherwise benefit from peer support in their local communities might not receive PSW visits. Hence, a quasi-experimental difference-in-differences (DID) study design was proposed in which the comparison group consists of those who are referred to the PSW programme but live outside of the four districts that comprise its current catchment area, and therefore do not have access to formal peer support. This design was modelled on a previous evaluation of community-based rehabilitation for people with severe mental disorders in India [43], and compares the change in number of rehospitalisations and hospital days among frequent users who have have access to a PSW (PSW+) to that of frequent users who do not have access to a PSW (PSW-).

Participants

Eligibility criteria

In order to be eligible for inclusion in the study, a service user must: (1) be age 18 years or older; (2) be a Ugandan national currently residing in-country; (3) have been diagnosed with either a mental disorder or epilepsy; (4) have had three or more inpatient stays at Butabika in the previous 24 months; (5) be referred to the peer support programme from one of five participating hospital wards at Butabika; (6) agree to participate in the study (assent) at baseline; and (7) provide either informed consent or guardian consent within the six-month follow-up period (Fig. 1). Although Butabika’s Alcohol and Drug Unit is not one of the hospital wards participating in this study, service users with mental or neurological disorders who also have co-morbid substance use disorders will not be excluded.

Fig. 1
figure 1

Flow chart for securing informed consent

Selection

Over a six-month recruitment period, staff from the five participating hospital wards will refer frequent users of inpatient care by completing referral forms with users’ demographic information, contact details, diagnoses, and admissions histories, extracted from patient records. As making referrals represents an additional unpaid administrative burden for already overstretched Butabika staff, Brain Gain II will offer a performance-based incentive to ward staff of 10,000 UGX (approximately $2.50 USD equivalent) for each referral form that is filled out completely and accurately. The Monitoring and Evaluation (M&E) Officer will review each form to confirm data quality before authorising a cash payment to be made directly to the staff member. Referral forms of sufficient quality will then be used to identify potential patients who meet the study criteria. Patients residing in Kampala, Jinja, Wakiso and Mukono will be eligible to receive the intervention, and patients residing elsewhere will form a comparison group.

Consent

The M&E Officer will assign specialised PSWs called “M&E Buddies” to visit potential participants on the hospital’s wards. M&E Buddies receive enhanced training in research procedures and ethics, including the use of the University of California, San Diego Brief Assessment of Capacity to Consent, a structured tool designed to assist research workers in assessing capacity to consent [44]. During the visit, a M&E Buddy will review the study information sheet and consent form with the participant in either English or Luganda and answer any questions.

All potential participants must assent before baseline data can be collected. Informed consent must also be secured in order for this data to be included in the evaluation. If the M&E Buddy suspects that a potential participant may not have capacity to consent, a guardian can consent as a substitute. Alternatively, a potential participant can provide informed consent at any point over a six-month follow-up period if he or she regains capacity. This provision is made to ensure that potential participants are empowered as much as possible to make their own decisions regarding participation. The process of securing informed consent is illustrated in Fig. 1.

Peer support

All participants receive standard adult care and may have some contact with PSWs via the Recovery College. Additionally, participants in the PSW+ group receive face-to-face visits from a trained PSW. Each participant in the PSW+ group is assigned a PSW from a nearby community by an administrator in the Peer Support Office. Peer support visits are offered for up to six months after a PSW is assigned to a user. A recipient of peer support will be visited by a PSW at least once on the ward before discharge, and at least three times after discharge. For particularly vulnerable cases (i.e. three or more inpatient stays in the past 12 months), up to six visits can be made. The PSW visits occur per the recipient’s preference, either at home, at a meeting point in the community, or at the hospital when the recipient returns for outpatient visits. The carer may also be engaged in peer support visits, where possible.

The peer support visits delivered by PSWs are flexible and unstructured, and may consist of any or all of the following:

  • Befriending (social contact, supportive listening and encouragement);

  • Role-modelling (sharing personal experiences of illness and recovery);

  • Psychoeducation (education on recovery principles);

  • Problem-solving (discussing current challenges and possible solutions, liasing with providers to resolve issues with medical and social care as needed).

The qualifications, training, supervision and compensation of PSWs as well as quality assurance for peer support visits are detailed in Table 1.

Table 1 Brain Gain II Peer Support Workers

Outcomes

For variables to be included in the primary analysis, Table 2 summarises the time-points for assessment, the form used and the data collector responsible.

Table 2 Assessment of Outcome Variables and Confounders for Primary Analysis

There are two primary outcomes for this study, both related to change in frequent users’ utilisation of inpatient care at Butabika. The first is change in the number of hospital days over the previous six months. The second is change in the number of rehospitalisations over the previous six months. The index hospitalisation during which the participant was recruited into the study is excluded from both calculations. The study flow chart in Fig. 2 further illustrates the sequence of assessments in each group.

Fig. 2
figure 2

Flow chart for quasi-experimental study

Hospital days

Hospital days are the number of days spent in psychiatric inpatient care at Butabika over the previous six months. In order to calculate the number of hospital days, dates of entry and departure are extracted from the patient file at the point of referral and six months from referral. An entry may be the result of formal admission or return to the hospital’s premises following an escape. Similarly, a departure may be the result of formal discharge or escape from the hospital’s premises.

Number of rehospitalisations

Number of rehospitalisations is the overall number of inpatient stays at Butabika over the previous six months. An inpatient stay is defined as a period of time spent in psychiatric inpatient care at Butabika and is also derived from the entry and departure data extracted from the patient file at the point of referral and six months from referral.

Potential confounders

Our initial working model described in Fig. 3 suggests four confounders from previous research on risk of readmission in other sub-Saharan African countries [6, 45], two from HICs [46,47,48,49,50], and four proposed by the investigators: baseline values for the number of rehospitalisations and hospital days; baseline values for disability and family support; demographic factors, including gender, age, marital status, education level and employment; and diagnosis.

Fig. 3
figure 3

Initial working model

However, DID is designed to adjust for time-invariant and group-invariant confounders [51]. Age, gender, diagnosis, education level, employment and marital status are unlikely to change substantially between groups within the six-month follow-up period. We also observe cautions by Glymour et al. (2005) regarding adjustment for baseline measures of outcome variables (hospital days and rehospitalisations) [52]. Therefore, we plan to include only the remaining two proposed confounders (disability and family support) as pre-specified covariates in the adjusted analysis.

The M&E Buddy collects data on these two potential confounders by administering a baseline form to the study participant while he or she is still on the hospital wards [Table 2]. For those in PSW+ group, it is completed before any PSW visits take place.

Disability

Disability score is assessed using the 12-item World Health Organisation Disability Assessment Scale 2.0 (WHODAS 2.0) [53]. WHODAS originated as a tool for use in psychiatric inpatient settings. WHODAS 2.0 has been validated in diverse cultural contexts and settings, and is considered to be an acceptable cross-cultural measurement tool [53,54,55]. A Luganda version has been developed and used in primary care settings in Uganda by the PRIME research project [56]. We will use the unweighted “simple scoring” technique manualised by Ustun et al. (2010) to calculate disability scores from the WHODAS Likert scales.

Family support

Family support is measured using two separate three-item Likert scales: one for family’s attitude, and one for family’s willingness to help. Face validity of these scales was examined during the development of the Mental health Information aND Monitoring and Evaluation (MIND ME) Nigeria toolkit, and these measures have since been integrated into routine data collection for monitoring and evaluation of three Nigerian mental health programmes [57,58,59]. As with the “simple scoring” technique for WHODAS, we plan to sum these Likert scales in order to produce overall family support scores.

Bias

Although this is not a blinded study, neither M&E Buddies, PSWs or the Peer Support Office Administrator responsible for assigning PSWs have access to participants’ admissions data. Admissions data are extracted from hospital records by ward staff at baseline and by a M&E Officer at follow-up [Table 2]. To reduce the risk of bias in the collection of additional data, a participant’s M&E Buddy will not also be assigned as his or her PSW.

Study size

Brain Gain II aims to enrol 180 users into peer support over six months. Assuming at least a 15% refusal rate and 15% loss-to-follow-up [60], we expect a maximum of 126 participants to complete follow-up in the PSW+ group. With this estimate in mind, we conducted a sensitivity analysis exploring a variety of different scenarios relating to power, effect size and intraclass correlation, allowing for both balanced and unbalanced samples, at 0.05 alpha (see Additional file 2). Results suggest it is possible to detect a clinically meaningful effect (20% as per Cohen [1998]) with 80% power, or a larger effect (30%) with 90% power, if an additional 87 users are followed up in the PSW- group [61]. We therefore aim to recruit at least 129 users to the PSW+ and 101 to the PSW- group.

Analysis

A detailed a priori statistical analysis plan has been drafted in consultation with a biostatistician and covers the baseline, primary and exploratory analyses described below. The plan will be finalised before any data are analysed and is available from the authors upon request. Any departures from the statistical analysis plan must be itemised and clearly justified in study reports.

We will use Stata/SE v15.1 for statistical analysis. All analyses will be performed at the level of the individual. Significance tests will be carried out with two-sided alpha of 0.05, and results will be reported with 95% confidence intervals.

Baseline

Descriptive statistics will be tabulated at baseline for both study groups (PSW+ and PSW-) [Table 3]. The mean, standard deviation and number of observations will be presented for all continuous variables. Numbers and percentages will be presented for categorical variables. Significance tests will not be performed to test for baseline differences between the study groups, as DID already presumes that study groups are unequivalent at baseline [51].

Table 3 Baseline characteristics for descriptive analysis

Primary

The primary analyses will compare the six-month change in primary outcomes (hospital days and number of rehospitalisations) between the PSW+ and PSW- groups. We will carry out both intention-to-treat and per-protocol DID analyses for the two primary outcomes, presenting adjusted and unadjusted results [62]. To be included in the per-protocol analysis, participants in the PSW+ arm must receive at least one recorded ward visit and three recorded community visits during the six-month follow-up period [62]. As described above, disability and family support are potential confounders and will be included as covariates in the adjusted analysis.

We will use multivariable linear regression unless the distribution is skewed, in which case we may consider a Poisson or negative binomial regression, or another appropriate method. The impact of the peer support programme on hospital days and rehospitalisations will be estimated through mixed effects models with a random effect to account for correlations among users with the same PSW and fixed effects for access to peer support (PSW+ group versus PSW-group), time (baseline versus follow-up) and the interaction between access to peer support and time. The interaction estimating the change from baseline to follow-up in the PSW+ group relative to change in the PSW- group is the key effect of interest. We will tabulate the results at follow-up and differences from baseline by group with corresponding 95% confidence intervals.

Exploratory

Further exploratory analyses will be carried out to help contextualise the results of the main analyses and generate hypotheses for future testing. As primary data collection from the PSW- group is not possible at six-month follow-up, exploratory analyses are limited to the PSW+ group.

Disability, service satisfaction and family support outcomes

Likert scale data on disability, satisfaction with services and family support will be collected from the PSW+ group by M&E Buddies at both baseline and six-month follow-up. For both disability and family support, we will calculate summary scores at baseline and follow-up in the PSW+ group [53], then perform a one-sample paired t-test. If the data distribution is skewed, we will consider the Wilcoxon signed-rank test or another non-parametric alternative. For the five-item Likert scale question on service satisfaction, we will use chi-squared tests to test the significance of the difference between baseline and follow-up in the PSW+ group.

Other psychosocial outcomes

Retrospective data on a number of other psychosocial outcomes will be collected from the PSW+ group by M&E Buddies at six-month follow-up. These are categorical variables labelled “improved”, “no change” or “worsened”. We will present descriptive statistics showing proportion of participants in the PSW+ group who reported “improved” or “worsened” outcomes, for the following:

  • Marriage or romantic relationship

  • Parenthood

  • Relationship with main caregiver

  • Relationship with other family members

  • Relationship with hospital staff (not PSWs)

  • Social relationships (e.g. friends, neighbours)

  • Work or income

  • Education or training

  • Housing

  • Hobbies or recreation

  • Physical health

Missing data

We expect attrition to be low, as the primary outcome data are derived from hospital records, meaning no follow-up contact with participants is required for the main analyses. The most likely reason for missing outcome data is loss of hospital records, which is not expected to affect the study groups differentially.

As per Schafer’s (1999) guidance, we will consider up to 5 % missing data to be inconsequential [63]. However, if more than 5 % of data is missing, then we will select appropriate principled missing data methods (e.g. multiple imputation), taking into consideration the data distribution and the mechanism, rate and pattern of missing data [64].

Data quality

Although this study was not resourced to enable double data entry, we will undertake a number of other precautions to improve data quality. As described above, forms used for data collection are routinely checked for quality by the Brain Gain II M&E Officer, and performance-based incentives are also offered to staff for complete and accurate extraction of admissions data. Data validation rules are programmed into the spreadsheet used for data entry. We will carry out additional range and consistency checks during data cleaning.

At the conclusion of the six-month follow-up period, we will also perform a data quality audit. The primary investigator will sample every fourth participant file and check each paper form against a checklist for missing, illegible or illogical data. Data quality issues will be disaggregated by type and tabulated by data collector and question, to identify any common patterns. At the stage of data analysis, we will use either box-plots or z-scores to identify outliers. Where outliers are clearly the result of spurious data, corrections will be made if possible; otherwise, outliers resulting from spurious data may be treated as missing data.

Ethics

This study was approved as part of a broader evaluation protocol for Brain Gain II. Institutional approval was received from the Research and Training Committee of the Butabika National Referral Hospital in Uganda. Ethics approval was secured from the London School of Hygiene and Tropical Medicine Research Ethics Committee in the United Kingdom (Ref 10,705) and Mengo Hospital Research and Ethics in Uganda (Ref 906/7). The evaluation protocol was also approved by the Uganda National Council of Science and Technology (Ref HS12ES). Additional details on the ethical considerations and procedures for this study are available from the study authors upon request.

Discussion

This study will contribute to the evaluation of one of the first formal peer support programmes to be established in a LMIC. Given the high cost of inpatient care, the outcomes are particularly relevant to mental health policy in Uganda and other LMICs, where most government expenditure on mental health continues to be spent on psychiatric hospitals [23].

The use of a quasi-experimental study design is an improvement over previous evaluations of formal peer support in LMICs. Although formal peer support programmes have been established in statutory services in other LMICs such as China [65] and India [66], to the best of our knowledge, only one has published a quantitative evaluation of user-level outcomes. These outcomes were limited to change in mood and social communication skills, assessed retrospectively at a single time-point, with no comparison group [65, 67].

This is also one of remarkably few examples in which the manpower and unique expertise of people with lived experience of mental and neurological disorders is harnessed for the purposes of conducting evaluation research in a LMIC. A systematic review published in 2016 identified only one previous example; it came from Brazil, an upper-middle income country, and users were not involved until data had already been collected [68, 69]. A 2017 survey on psychosocial disabilities and barriers to participation in North India may have involved data collectors with psychosocial disabilities, although this is unclear from the study’s text [67, 70]. While engagement of M&E Buddies in data collection is desirable from an inclusion perspective, and may be more sustainable than relying on external evaluators, the use of M&E Buddies has not yet been tested in this context. Future publications will report not only on the outcomes of peer support in Uganda, but also on learning from the engagement of peers in the conduct of this study.

Unfortunately, a randomized control trial was not deemed acceptable for the purposes of this study. Indeed, researchers in HICs have pointed out that randomisation may be in opposition to the principles of self-determination embraced by peer support programmes [71], though some have carried out successful randomised-controlled trials nonetheless. DID is designed to control for time-invariant and group-invariant confounders, but does not entirely eliminate the possibility of confounding or other types of bias. As noted by Wing, Simon and Bello-Gomez (2018): “The DID design is not a perfect substitute for randomized experiments, but it often represents a feasible way to learn about causal relationships” (pp.453).

The study design outlined in this protocol introduces a risk of bias, as the two study groups differ by district of residence. There is a possibility that unmeasured time- and group-variant factors may differ between districts and confound the relationship between peer support and use of inpatient care. Peer support is only available in four districts of the relatively prosperous Central and Eastern regions near the country’s capital city, Kampala, where Butabika is located. Further, this is not a blinded study, though PSWs responsible for delivering the intervention do not collect study data from their assigned peers, and neither PSWs, M&E Buddies nor the Peer Support Office Administrator have access to admissions data.

In addition, the outcomes compared between groups are limited to those which can be assessed using the hospital’s routinely collected data. Loss-to-follow-up after discharge from inpatient care is a significant issue for Butabika, which PSWs help to remedy through regular visits to the community. The comparison group does not have contact with PSWs after discharge. Therefore, stakeholders deemed it unfeasible to collect primary data from a significant number of participants in the comparison group. The Indian evaluation upon which this study is based faced similar challenges; outcomes could not be assessed in about a third of all participants in the comparison group [43].

Finally, it is worth acknowledging that formal peer support can be challenging to implement [72]. For example, there is a risk that formalisation of peer support roles may actually reinforce hierarchical relationships in statutory services and ultimately undermine core peer support values [73]. At the same time, PSWs may be expected to serve as “carriers of a recovery culture into mental health systems” (Ibrahim 2019, n.page), resulting in conflict between PSWs and organisations resistant to change [72]. These are tricky relationships for PSWs to negotiate, even in relatively well-resourced settings. Meanwhile, resource limitations have been identified as major barriers to service user involvement in mental health systems strengthening in Uganda [74] and in LMICs more broadly [75]. While the Brain Gain I pilot demonstrated that it is feasible to deliver formal peer support in Uganda [31], there is always the possibility of “implementation failure” (Patton 2008, pp. 310) leading to null results [76]. In this case, our multi-method, Theory of Change-driven approach to the broader programme evaluation (described elsewhere [34, 35]) may help to identify “what went wrong” on the anticipated pathway of change and make targeted recommendations for future implementation.

In conclusion, this is a quasi-experimental DID study subject to a number of different constraints and potential biases, which should be followed up with more robust research, assessing more outcomes with locally validated measures, and ideally using a randomised design. Due caution will need to be taken in the interpretation of results. However, given the paucity of research currently available from LMICs, this study represents a crucial next step toward the development of a global evidence base for peer support.

Availability of data and materials

Not applicable.

Abbreviations

BELL:

Butabika-East London National Health Service Foundation Trust Link

DID:

Difference-in-differences

ELFT:

East London National Health Service Foundation Trust

HICs:

High-Income Countries

LMICs:

Low- and Middle-Income Countries

M&E:

Monitoring and Evaluation

MIND ME:

Mental Health Information and Monitoring and Evaluation

PSW-:

No Access to Peer Support

PSW:

+ Access to Peer Support

PSWs:

Peer Support Workers

STROBE:

Strengthening the Reporting of Observational Studies in Epidemiology

UGX:

Uganda Shillings

USD:

United States Dollars

WHO:

World Health Organisation

WHODAS:

World Health Organisation Disability Assessment Schedule

References

  1. Botha UA, Koen L, Joska JA, Parker JS, Horn N, Hering LM, Oosthuizen PP. The revolving door phenomenon in psychiatry: comparing low-frequency and high-frequency users of psychiatric inpatient services in a developing country. Soc Psychiatry Psychiatr Epidemiol. 2010;45(4):461–8.

    Article  PubMed  Google Scholar 

  2. Oyffe I, Kurs R, Gelkopf M, Melamed Y, Bleich A. Revolving-door patients in public psychiatric Hospital in Israel: cross sectional study. Croatian Medical Journal. 2009;50(6):575–82.

    Article  PubMed  PubMed Central  Google Scholar 

  3. Kent S, Fogarty M, Yellowlees P. A review of studies of heavy users of psychiatric services. Psychiatr Serv. 1995;46(12):1247–53.

    Article  CAS  PubMed  Google Scholar 

  4. Ndetei DM, Khasakhala L, Maru H, Pizzo M, Mutiso V, Ongecha-Owuor FA, Kokonya DA. Clinical epidemiology in patients admitted at Mathari psychiatric hospital, Nairobi, Kenya. Social Psychiatry and Psychiatric Epidemiology. 2008;43(9):736–42.

    Article  PubMed  Google Scholar 

  5. Ucok A, Polat A, Cakir S, Genc A. One year outcome in first episode schizophrenia. Predictors of relapse. Eur Arch Psychiatry Clin Neurosci. 2006;256(1):37–43.

    Article  PubMed  Google Scholar 

  6. Yussuf AD, Kuranga SA, Balogun OR, Ajiboye PO, Issa BA, Adegunloye O, Parakoyi MT. Predictors of psychiatric readmissions to the psychiatric unit of a tertiary health facility in a Nigerian city - a 5-year study. Afr J Psychiatry. 2008;11(3):187–90.

    Article  CAS  Google Scholar 

  7. Gbiri CA, Badru FA, Ladapo HT, Gbiri AA. Socio-economic correlates of relapsed patients admitted in a Nigerian mental health institution. Int J Psychiatry Clin Pract. 2011;15(1):19–26.

    Article  PubMed  Google Scholar 

  8. Barekatain M, Maracy MR, Hassannejad R, Hosseini R: Factors Associated with Readmission of Patients at a University Hospital Psychiatric Ward in Iran. Psychiatry J 2013, 2013:685625.

  9. Gastal FL, Andreoli SB, Quintana MIS, Gameiro MA, Leite SO, McGrath J. Predicting the revolving door phenomenon among patients with schizophrenic, affective disorders and non-organic psychoses. Rev Saude Publica. 2000;34:280–5.

    Article  CAS  PubMed  Google Scholar 

  10. Ezenduka C, Ichoku H, Ochonma O. Estimating the costs of psychiatric hospital services at a public health facility in Nigeria. The Journal of Mental Health Policy and Economics. 2012;15(3):139–48.

    PubMed  Google Scholar 

  11. Ådnanes M, Melby L, Cresswell-Smith J, Westerlund H, Rabbi L, Dernovšek MZ, Šprah L, Sfetcu R, Straßmayr C, Donisi V. Mental health service users' experiences of psychiatric re-hospitalisation - an explorative focus group study in six European countries. BMC Health Serv Res. 2018;18(1) 516–516.

  12. HRW: "Treated Worse than Animals": Abuses against Women and Girls with Psychosocial or Intellectual Disabilities in Institutions in India. In.: Human Rights Watch; 2014.

  13. HRW: Living in Hell: Abuses against People with Psychosocial Disabilities in Indonesia. In. Bahasa, Indonesia: Human Rights Watch; 2016.

  14. HRW: "Like a Death Sentence": Abuses against Persons with Mental Disabilities in Ghanda. In.: Human Rights Watch; 2012.

  15. MDAC: Breaking Point: Findings from a Visit to Butabika National Mental Referral Hospital, Uganda. In.: Mental Disability Advocacy Centre; 2017.

  16. MDAC: The Right to Legal Capacity in Kenya. In.: Mental Disability Advocacy Centre, Mental Health Uganda; 2014.

  17. Drew N, Funk M, Tang S, Lamichhane J, Chávez E, Katontoka S, Pathare S, Lewis O, Gostin L, Saraceno B. Human rights violations of people with mental and psychosocial disabilities: an unresolved global crisis. Lancet. 2011;378(9803):1664–75.

    Article  PubMed  Google Scholar 

  18. Chinman M, George P, Dougherty RH, Daniels AS, Ghose SS, Swift A, Delphin-Rittmon ME. Peer support services for individuals with serious mental illnesses: assessing the evidence. Psychiatr Serv. 2014;65(4):429–41.

    Article  PubMed  Google Scholar 

  19. Davidson L, Bellamy C, Guy K, Miller R. Peer support among persons with severe mental illnesses: a review of evidence and experience. World Psychiatry. 2012;11(2):123–8.

    Article  PubMed  PubMed Central  Google Scholar 

  20. Simpson EL, House AO: Involving users in the delivery and evaluation of mental health services: systematic review. BMJ (Clinical research ed) 2002, 325(7375):1265.

    Article  PubMed  PubMed Central  Google Scholar 

  21. Solomon P. Peer support/peer provided services underlying processes, benefits, and critical ingredients. Psychiatric Rehabilitation Journal. 2004;27(4):392.

    Article  PubMed  Google Scholar 

  22. Sledge WH, Lawless M, Sells D, Wieland M, O'Connell MJ, Davidson L. Effectiveness of peer support in reducing readmissions of persons with multiple psychiatric hospitalizations. Psychiatr Serv. 2011;62(5):541–4.

    Article  PubMed  Google Scholar 

  23. WHO: Mental Health Atlas 2017. In. Geneva, Switzerland; World Health Organisation 2018.

  24. Repper J, Carter T. A review of the literature on peer support in mental health services. J Ment Health. 2011;20(4):392–411.

    Article  PubMed  Google Scholar 

  25. Slade M, Amering M, Farkas M, Hamilton B, O'Hagan M, Panther G, Perkins R, Shepherd G, Tse S, Whitley R. Uses and abuses of recovery: implementing recovery-oriented practices in mental health systems. World Psychiatry. 2014;13(1):12–20.

    Article  PubMed  PubMed Central  Google Scholar 

  26. Slade M, Amering M, Oades L. Recovery: an international perspective. Epidemiologia e Psichiatria Sociale. 2008;17(2):128–37.

    Article  PubMed  Google Scholar 

  27. Davidson L, Chinman M, Kloos B, Weingarten R, Stayner D, Tebes JK. Peer support among individuals with severe mental illness: a review of the evidence. Clin Psychol Sci Pract. 1999;6(2):165–87.

    Article  Google Scholar 

  28. Davidson L, Chinman M, Sells D, Rowe M. Peer support among adults with serious mental illness: a report from the field. Schizophr Bull. 2006;32(3):443–50.

    Article  PubMed  PubMed Central  Google Scholar 

  29. Baillie D, Hall C: Brain Gain: The benefits of working as Peer Support Workers for Service Users in Urban Uganda. In: Centre for Global Mental Health Forum: Sustainable development through global action. King's College Institute of Psychiatry, London, UK; 2013.

  30. Baillie D, Aligawesa M, Birabwa-Oketcho H, Hall C, Kyaligonza D, Mpango R, Mulimira M, Boardman J. Diaspora and peer support working: benefits of and challenges for the Butabika–East London link. BJPsych International. 2015;12(1):10–3.

    Article  PubMed  PubMed Central  Google Scholar 

  31. Hall C, Baillie D, Atukunda J: Brain Gain in Uganda: A Case Study of Peer Working as an Adjunct to Statutory Mental Health Care in a Low-Income Country. In: The Palgrave Handbook of Sociocultural Perspectives on Global Mental Health. edn. Edited by White R, Jain S, Orr D, Read U. London: Palgrave Macmillan; 2017: 633–655.

    Chapter  Google Scholar 

  32. Ev E, Altman DG, Egger M, Pocock SJ, Gøtzsche PC, Vandenbroucke JP. strengthening the reporting of observational studies in epidemiology (STROBE) statement: guidelines for reporting observational studies. BMJ. 2007;335(7624):806–8.

    Article  Google Scholar 

  33. Reeves BC, Gaus W: Guidelines for reporting non-randomised studies. Forschende Komplementarmedizin und klassische Naturheilkunde = Research in complementary and natural classical medicine 2004, 11 Suppl 1:46–52.

    Article  Google Scholar 

  34. Ryan G, Baillie D: MORE brain gain: evaluating the impact of peer support work in Uganda. In: King’s Health Partners Global Mental Health and Psychosocial Support Conference. London, UK; 2017.

  35. Mpango R, Ryan GK: Brain gain: generating evidence of effectiveness to inform scale-up of peer support in Kampala, Uganda. In: Tropical Health Education Trust Annual Conference. London, UK; 2016.

  36. Butabika National Referral Hospital [http://www.emerald-project.eu/the-group/consortium/butabika-national-referral-and-teaching-mental-hospital/].

  37. Hall C, Kapadia Y, Ryan GK, Mulimira M, Baillie D: Brain Gain 2: Challenges and Lessons in the Development of Recovery-Oriented Mental Health Services in Uganda. In: Global Mental Health Conference: Promoting positive mental health through a community development approach in a global mental health context. Brighton, UK; 2015.

  38. Whitley R, Shepherd G, Slade M. Recovery colleges as a mental health innovation. World Psychiatry. 2019;18(2):141–2.

    Article  PubMed  PubMed Central  Google Scholar 

  39. Group SS. Recovery in Uganda: themes from recovery listening events. In. Sharing Stories Venture: Preston; 2015.

    Google Scholar 

  40. De Silva MJ, Breuer E, Lee L, Asher L, Chowdhary N, Lund C, Patel V. Theory of change: a theory-driven approach to enhance the Medical Research Council's framework for complex interventions. Trials. 2014;15(1):267.

    Article  PubMed  PubMed Central  Google Scholar 

  41. Weiss CHJ, Nateci C: Nothing as practical as good theory: Exploring theory-based evaluation for comprehensive community initiatives for children and families. 1995, 1:65–92.

  42. Chibanda D, Verhey R, Munetsi E, Cowan FM, Lund C. Using a theory driven approach to develop and evaluate a complex mental health intervention: the friendship bench project in Zimbabwe. Int J Ment Heal Syst. 2016;10(1):16.

    Article  Google Scholar 

  43. Chatterjee S, Patel V, Chatterjee A, Weiss HA. Evaluation of a community-based rehabilitation model for chronic schizophrenia in rural India. Br J Psychiatry. 2003;182(1):57–62.

    Article  PubMed  Google Scholar 

  44. Jeste DV, Palmer BW, Appelbaum PS, et al. A new brief instrument for assessing decisional capacity for clinical research. Arch Gen Psychiatry. 2007;64(8):966–74.

    Article  PubMed  Google Scholar 

  45. Niehaus DJ, Koen L, Galal U, Dhansay K, Oosthuizen PP, Emsley RA, Jordaan E. Crisis discharges and readmission risk in acute psychiatric male inpatients. BMC Psychiatry. 2008;8:44.

    Article  PubMed  PubMed Central  Google Scholar 

  46. Lorine K, Goenjian H, Kim S, Steinberg AM, Schmidt K, Goenjian AK. Risk factors associated with psychiatric readmission. J Nerv Ment Dis. 2015;203(6):425–30.

    Article  PubMed  Google Scholar 

  47. van der Post LF, Mulder CL, Peen J, Visch I, Dekker J, Beekman AT. Social support and risk of compulsory admission: part IV of the Amsterdam study of acute psychiatry. Psychiatr Serv. 2012;63(6):577–83.

    Article  PubMed  Google Scholar 

  48. Swett C. Symptom severity and number of previous psychiatric admissions as predictors of readmission. Psychiatr Serv. 1995;46(5):482–5.

    Article  CAS  PubMed  Google Scholar 

  49. Bobier C, Warwick M. Factors associated with readmission to adolescent psychiatric care. The Australian and New Zealand Journal of Psychiatry. 2005;39(7):600–6.

    Article  PubMed  Google Scholar 

  50. Figueroa R, Harman J, Engberg J. Use of claims data to examine the impact of length of inpatient psychiatric stay on readmission rate. Psychiatr Serv. 2004;55(5):560–5.

    Article  PubMed  Google Scholar 

  51. Wing C, Simon K, Bello-Gomez RA. Designing difference in difference studies: best practices for public health policy research. Annu Rev Public Health. 2018;39:453–69.

    Article  PubMed  Google Scholar 

  52. Glymour MM, Weuve J, Berkman LF, Kawachi I, Robins JM. When is baseline adjustment useful in analyses of change? An example with education and cognitive change. Am J Epidemiol. 2005;162(3):267–78.

    Article  PubMed  Google Scholar 

  53. WHO: Measuring Health and Disability: Manual for WHO Disability Assessment Schedule (WHODAS 2.0). In. Edited by Ustun T, Kostanjsek N, Chatterji S, Rehm J. Geneva: World Health Organisation; 2010.

  54. Üstün TB, Chatterji S, Villanueva M, Bendib L, Çelik C, Sadana R, Valentine N, Ortiz J. Tandon a. WHO multi-country survey study on health and responsiveness: Salomon JJGWHO; 2001.

    Google Scholar 

  55. Habtamu K, Alem A, Medhin G, Fekadu A, Dewey M, Prince M, Hanlon C. Validation of the World Health Organization disability assessment schedule in people with severe mental disorders in rural Ethiopia. Health Qual Life Outcomes. 2017;15(1) 64–64.

  56. De Silva MJ, Rathod SD, Hanlon C, Breuer E, Chisholm D, Fekadu A, Jordans M, Kigozi F, Petersen I, Shidhaye R, et al. Evaluation of district mental healthcare plans: the PRIME consortium methodology. Br J Psychiatry. 2018;208(s56):s63–70.

    Article  Google Scholar 

  57. Eaton J, Gureje O, De Silva M, Sheikh TL, Ekpe EE, Abdulaziz M, Muhammad A, Akande Y, Onukogu U. Onyuku TJIjomhs: a structured approach to integrating mental health services into primary care: development of the mental health scale up Nigeria intervention (mhSUN). Int J Ment Heal Syst. 2018;12(1):11.

    Article  Google Scholar 

  58. Ryan G, De Silva M, Terver JS, Ochi OP, Eaton J. Information systems for global mental health. Lancet Psychiatry. 2015;2(5):372–3.

    Article  PubMed  Google Scholar 

  59. Ryan G, DeSilva M, Eaton J, ode P, Nwefoh E, Cohen a: formative research to scale up clinical monitoring and evaluation for mental health services in Benue state, Nigeria. In: Centre for Global Mental Health Forum. London; 2013.

  60. Patel V, Weiss HA, Chowdhary N, Naik S, Pednekar S, Chatterjee S, De Silva MJ, Bhat B, Araya R, King M et al: Effectiveness of an intervention led by lay health counsellors for depressive and anxiety disorders in primary care in Goa, India (MANAS): a cluster randomised controlled trial. The Lancet, 376(9758):2086–2095.

    Article  Google Scholar 

  61. Cohen J. Statistical power analysis in the behavioral sciences. 2nd ed. Mahwah: Lawrence Erlbaum Associates; 1988.

  62. Gertler PJ, Martinez S, Premand P, Rawlings LB, Vermeersch CMJ. Impact evaluation in practice. In. The World Bank: Washington, D.C; 2011.

    Google Scholar 

  63. Schafer JL. Multiple imputation: a primer. Stat Methods Med Res. 1999;8(1):3–15.

    Article  CAS  PubMed  Google Scholar 

  64. Dong Y, Peng C-YJ. Principled missing data methods for researchers. Springerplus. 2013;2(1) 222–222.

  65. Fan Y, Ma N, Ma L, Xu W, Steven Lamberti J, Caine ED. A community-based peer support service for persons with severe mental illness in China. BMC Psychiatry. 2018;18(1):170.

    Article  PubMed  PubMed Central  Google Scholar 

  66. Pathare S, Kalha J, Krishnamoorthy S. Peer support for mental illness in India: an underutilised resource. Epidemiology and Psychiatric Sciences. 2018:1–5.

  67. Ryan G, Semrau M, Nkurunungi E, Mpango R. Service user involvement in global mental health: what have we learned from recent research in low- and middle-income countries? Current Opinion in Psychiatry; 2019.

    Book  Google Scholar 

  68. Semrau M, Lempp H, Keynejad R, Evans-Lacko S, Mugisha J, Raja S, Lamichhane J, Alem A, Thornicroft G, Hanlon C. Service user and caregiver involvement in mental health system strengthening in low- and middle-income countries: systematic review. BMC Health Serv Res. 2016;16(1):79.

    Article  PubMed  PubMed Central  Google Scholar 

  69. Furtado JP. Campos RO: [participation, knowledge production, and evaluative research: participation by different actors in a mental health study]. Cadernos de saude publica. 2008;24(11):2671–80.

    Article  PubMed  Google Scholar 

  70. Mathias K, Pant H, Marella M, Singh L, Murthy G, Grills N. Multiple barriers to participation for people with psychosocial disability in Dehradun district, North India: a cross-sectional study. BMJ Open. 2018;8(2):e019443.

    Article  PubMed  PubMed Central  Google Scholar 

  71. Landers GM, Zhou M. An analysis of relationships among peer support, psychiatric hospitalization, and crisis stabilization. Community Ment Health J. 2011;47(1):106–12.

    Article  PubMed  Google Scholar 

  72. Ibrahim N, Thompson D, Nixdorf R, Kalha J, Mpango R, Moran G, Mueller-Stierlin A, Ryan G, Mahlke C, Shamba D, et al. A systematic review of influences on implementation of peer support work for adults with mental health problems. Soc Psychiatry Psychiatr Epidemiol. 2019.

  73. Faulkner A, Kalathil JJLT. The freedom to be, the chance to dream: preserving user-led peer support in mental health; 2012.

    Google Scholar 

  74. Mugisha J, Hanlon C, Knizek BL, Ssebunnya J, Vancampfort D, Kinyanda E, Kigozi F. The experience of mental health service users in health system strengthening: lessons from Uganda. Int J Ment Heal Syst. 2019;13(1):60.

    Article  Google Scholar 

  75. Lempp H, Abayneh S, Gurung D, Kola L, Abdulmalik J, Evans-Lacko S, Semrau M, Alem A, Thornicroft G, Hanlon CJE, et al. Service user and caregiver involvement in mental health system strengthening in low-and middle-income countries: a cross-country qualitative study. BMC Health Serv Res. 2018;27(1):29–39.

    CAS  Google Scholar 

  76. Patton MQ. Utlization-focused evaluation, 4th edition edn. Thousand Oaks, California: SAGE Publications, Inc.; 2008.

    Google Scholar 

Download references

Acknowledgments

We thank Professor Elizabeth Allen, Dr. Alex Cohen, Dr. Clare Tanton and Georgina Miguel Esponda at London School of Hygiene and Tropical Medicine, for reviewing previous drafts of this manuscript. We also wish to acknowledge colleagues from Butabika and ELFT, including the Brain Gain II peer support workers, who contributed to the design of the Brain Gain II Theory of Change and provided input into logistical considerations in the design of this study.

Funding

This research is funded through a Health Partnership Scheme award by the Tropical Health Education Trust to the Butabika-ELFT Link (“BELL”). The funder had no role in the design of the study or in writing this manuscript, and will have no role in the collection, analysis or interpretation of the data.

Author information

Authors and Affiliations

Authors

Contributions

GKR designed the study and wrote the original Brain Gain II protocol with supervision from JM, MJD and RM, and critical input from MK, DB, CH, EN and CN. SDR advised on the sample size calculation, and along with KMD assisted GKR in revising the manuscript for publication. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Grace K. Ryan.

Ethics declarations

Ethics approval and consent to participate

Informed consent must be confirmed via signature or thumb-print by the participant or guardian following the procedures described in this protocol, which were approved by the London School of Hygiene and Tropical Medicine Research Ethics Committee in the United Kingdom (Ref 10705), Mengo Hospital Research and Ethics in Uganda (Ref 906/7), and the Uganda National Council of Science and Technology in Uganda (Ref HS12ES).

Consent for publication

Not applicable.

Competing interests

The authors declare that they have no competing interests.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Additional file 1.

Brain Gain II Theory of Change map.

Additional file 2.

Sensitivity Analysis for Primary Outcomes.

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ryan, G.K., Kamuhiirwa, M., Mugisha, J. et al. Peer support for frequent users of inpatient mental health care in Uganda: protocol of a quasi-experimental study. BMC Psychiatry 19, 374 (2019). https://doi.org/10.1186/s12888-019-2360-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s12888-019-2360-8

Keywords