Skip to main content
  • Study protocol
  • Open access
  • Published:

The PULSAR primary care protocol: a stepped-wedge cluster randomized controlled trial to test a training intervention for general practitioners in recovery-oriented practice to optimize personal recovery in adult patients



General practitioners (GPs) in Australia play a central role in the delivery of mental health care. This article describes the PULSAR (Principles Unite Local Services Assisting Recovery) Primary Care protocol, a novel mixed methods evaluation of a training intervention for GPs in recovery-oriented practice. The aim of the intervention is to optimize personal recovery in patients consulting study GPs for mental health issues.


The intervention mixed methods design involves a stepped-wedge cluster randomized controlled trial testing the outcomes of training in recovery-oriented practice, together with an embedded qualitative study to identify the contextual enablers and challenges to implementing recovery-oriented practice. The project is conducted in Victoria, Australia between 2013 and 2017. Eighteen general practices and community health centers are randomly allocated to one of two steps (nine months apart) to start an intervention comprising GP training in the delivery of recovery-oriented practice. Data collection consists of cross-sectional surveys collected from patients of participating GPs at baseline, and again at the end of Steps 1 and 2. The primary outcome is improvement in personal recovery using responses to the Questionnaire about the Process of Recovery. Secondary outcomes are improvements in patient-rated measures of personal recovery and wellbeing, and of the recovery-oriented practice they have received, using the INSPIRE questionnaire, the Warwick-Edinburgh Mental Well-being Scale, and the Kessler Psychological Distress Scale. Participant data will be analyzed in the group that the cluster was assigned to at each study time point. Another per-protocol dataset will contain all data time-stamped according to the date of intervention received at each cluster site. Qualitative interviews with GPs and patients at three and nine months post-training will investigate experiences and challenges related to implementing recovery-oriented practice in primary care.


Recovery-oriented practice is gaining increasing prominence in mental health service delivery and the outcomes of such an approach within the primary care sector for the first time will be evaluated in this project. If findings are positive, the intervention has the potential to extend recovery-oriented practice to GPs throughout the community.

Trial registration

Australian and New Zealand Clinical Trial Registry (ACTRN12614001312639). Registered: 8 August 2014.

Peer Review reports


Australian primary care and mental health

Most Australian medical general practitioners (GPs) work within privately owned general practices where they play an important direct role in the diagnosis and management of mental health problems [1]; around 13% of GP consultations are related to mental health, with depression cited as the most common illness [2, 3]. Use of GP services directly or through prescribing contributes substantially to the over $8 billion per annum cost of mental health services [4], and Australian GPs also have specific roles in determining eligibility to many of the specialist services that make up the rest of this cost [5]. Referrals for mental health, as a percent of total GP encounters, are growing through time with an annual average increase of 7% in the five years up to 2013/14 [6].

In the State of Victoria, Australia, the arrangement for mental health services reflects the hybrid national system of mixed private and public service delivery. Australian Medicare, intended as a universal health insurance scheme, provides Australian residents the right to assign specified benefits to private health providers, including GPs, based on items of care including consultations. Co-payments are unrestricted and practitioners can choose their location of practice. The majority of people in Victoria with mental health issues access mental health services via their GP [7]. State-run public sector specialist clinical mental health services, typically accessed by people with more severe mental illnesses, are block-funded. Of all third-party expenditure on mental health care delivery, 61.0% is through State and Territory Governments, 35.2% from the Commonwealth Government, the remainder from private insurance [8]. In Victoria, state-funded mental health care includes a substantial investment in the non-clinical community mental health support sector (run by non-government organizations) which provide programs that help individuals manage their own recovery and maximize their involvement in community life [7].

Recovery-oriented practice, REFOCUS and the PULSAR project

An approach to mental health care that has gained momentum over recent years is recovery-oriented practice, which involves facilitating a process of change through which individuals are supported to build and live fulfilling and meaningful lives, with or without the continuing presence of mental health issues [9, 10]. The meaning of the term ‘recovery’ in this context is distinguished from clinical recovery and has been summarized as:

A deeply personal, unique process of changing one’s attitudes, values, feelings, goals, skills and roles. It is a way of living a satisfying, hopeful and contributing life even with limitations caused by the illness. Recovery involves the development of new meaning and purpose in one’s life as one grows beyond the catastrophic effects of mental illness [11].

A paradigm shift towards recovery-oriented practice in specialist mental health service delivery is being embraced internationally [1214]. An evidence-based package of tools, developed [15] and trialed [16] by the REFOCUS Team (Institute of Psychiatry, King’s College London) and known as the REFOCUS Intervention, has been used to promote recovery-oriented practice in specialist mental health teams in the UK. In Australia, recovery-oriented practice has been endorsed through the Australian National Mental Health strategy from the early 2000s [17] with various efforts to promote a recovery framework made in the eight States and Territories [18]. While there is extensive work on recovery-oriented practice in specialist service delivery, a focus on personal recovery is less well established as influential in GP training and practice. Moreover, to our knowledge, no trial has yet been published in Australia or elsewhere that has examined whether interventions promoting recovery-oriented practice in primary care improve outcomes for patients. Given that the GP is the service provider most commonly consulted for mental health problems [19], this represents a critical gap.

The Principles Unite Local Services Assisting Recovery (PULSAR) Primary Care project is part of the broader PULSAR research program into approaches to promoting recovery-oriented practice. PULSAR involves adaptation of REFOCUS materials [15, 20] for Australian primary and specialist care [21]. For the PULSAR Primary Care project, the REFOCUS materials were adapted for Australian privately owned GP practices and government funded community health centers. Introduction of these principles and practices into primary care with associated research is part of the scope of the PULSAR project and described in this paper. It might be noted that although formal introduction of recovery-oriented practice to general practice may be innovative, some elements may not be entirely novel to GPs since recovery-oriented practice can be seen as having common ground with influential concepts in primary care including, for instance, Shared Decision Making [2224], Patient Centered Care [25], and a doctor–patient relationship characterized by Mutual Participation [24]. We also note at this point that in description of parallel PULSAR Secondary Care projects we refer to people who experience mental illness and engage with services as ‘consumers’ but in the Primary Care project described in this paper, the term ‘patient’ will be used as more representative of regular usage in this context.

This paper describes the PULSAR Primary Care study protocol. The protocol adheres to the SPIRIT (Standard Protocol Items: Recommendations for Interventional Trials) 2013 guidelines [26].

Aims and objectives

The aim of the PULSAR Primary Care study is to conduct a mixed methods evaluation of a training intervention for GPs in recovery-oriented practice. The purpose of the intervention is to optimize personal recovery in patients consulting project study GPs for mental health issues. The project employs an intervention mixed methods design [27], involving a stepped-wedge cluster randomized controlled trial (cRCT) of training in recovery-oriented practice, together with an embedded qualitative study of implementing recovery-oriented practice in primary care. Clusters are participating general practices and community health centers that employ GPs. The objective of the quantitative research is to examine whether adult patients of GPs and practices that have received training in recovery-oriented practice report greater personal recovery compared to other patients where practices have not received the intervention. The following research questions are addressed:

  1. 1.

    From pre- to post-intervention, do patients of intervention cluster GPs report greater improvements on measures of personal recovery compared with control group patient participants?

  2. 2.

    From pre- to post-intervention, do patients of intervention cluster GPs report greater improvements on measures of health and wellbeing status compared with control group patient participants?

  3. 3.

    From pre- to post-intervention, do patients of intervention cluster GPs report greater improvements on measures of perceived need for care and satisfaction with services compared with control group participants?

  4. 4.

    Are any changes in estimated service costs compatible with favorable health economic properties for the intervention?

  5. 5.

    What are the contextual enablers and challenges to implementing recovery-oriented practice in primary care settings?


Overall design

The intervention mixed methods design involves a stepped-wedge cluster randomized controlled trial testing the outcomes of training in recovery-oriented practice, together with an embedded qualitative study to identify the contextual enablers and challenges to implementing recovery-oriented practice within primary care settings. Eighteen general practices and community health centers were randomly allocated to one of two steps (9 months apart) to start an intervention comprising general practitioner training in the delivery of recovery-oriented practice. Data collection consists of cross-sectional surveys collected from patients of participating general practitioners at three time points: baseline, and again at the end of Steps 1 and 2 (see Fig. 1). A mixed methods design was chosen for a number of reasons, including that it would produce richer and more complete findings than that based on either the qualitative or quantitative approach alone. Combining these approaches can further explain and enhance the integrity of the project outcomes, so that a mixed methods design may assist with project credibility, particularly when multiple stakeholders are involved such as in the PULSAR project [28]. The use of qualitative and quantitative methods were integrated at different stages of the intervention trial [27]. Initially, qualitative methods were used in an exploratory manner to identify potential barriers to recovery-oriented training within primary care settings, and to inform the intervention design. Subsequently, quantitative methods are being used to then measure the effectiveness of the intervention. Following the intervention trial, qualitative methods will be used in an explanatory manner to gain a more nuanced understanding of GP experiences of implementing recovery-oriented practices and patient experiences of these practices [28]. Quantitative and qualitative data are collected from patient participants, but only qualitative data are collected from GP participants.

Fig. 1
figure 1

A stepped-wedge cluster randomized controlled trial in general practice and community health center primary care settings. Data collection from patients is planned to occur during the six-month periods at baseline, Step 1, and Step 2

A process evaluation was nested within the study design to collect qualitative, quantitative, and documentary data. The purpose of the process evaluation is to explain those factors that impinge on the study design and how processes and decisions during the course of the study could explain or elaborate on findings. Process evaluation is often missing from randomized controlled trials; however, the UK Medical Research Council has provided useful guidelines for bringing rigor to the conduct of such evaluations. The value of process evaluation within trials such as the PULSAR study is to assess fidelity and quality of implementation, clarify causal mechanisms and identify contextual factors associated with variation in outcomes [29] These guidelines inform the approach in the current study [29] which includes analysis of process of implementing a complex intervention design associated with the stepped-wedge approach, sampling responses, effect sizes, and organizational and contextual factors that impact the study processes and outcomes. The process evaluation draws on the findings of the UK REFOCUS process evaluation [30] which explored service users’ experience of receiving a complex pro-recovery intervention. Furthermore, Bhanbhro and colleagues’ [31] recent work confirmed the crucial importance of taking account of practitioner, trainer and organizational factors when implementing a training intervention to achieve practice change in diverse mental health settings [31].

The process evaluation will focus on the following:

  1. 1.

    Service users’ experience of the PULSAR training and PALS intervention

  2. 2.

    The implementation context

  3. 3.

    The research process and engagement

  4. 4.

    The impact and outcomes of the study.

Explanation for choice of comparators

The design was developed to combine the rigor of a cluster randomized trial with the pragmatic approach of the stepped wedge design to implement the intervention at all sites [3234]. Control sites are those that are yet to receive the intervention. Since all sites eventually receive the intervention, data from sites in control phases is to be compared with data from sites that have received the intervention. There are no study restrictions on the care provided in control phases.

Study setting

Recruitment of cluster sites and GPs was initially to be within the Monash Health catchment in Victoria, Australia [35]. However, challenges in recruiting GPs emerged when this geographical restriction was applied, so recruitment was extended to include any cluster site in the greater Monash Health region and adjacent areas. This encompasses the catchment areas of three Medicare Locals that overlap with the Monash Health catchment. Medicare Locals were organizations established to coordinate the delivery of primary health care within bounded geographical regions so as to address local health care priorities and improve access to primary care. These were decommissioned during the project (30 June 2015) and replaced with Primary Health Networks, which created challenges for ongoing engagement with GPs.

The study setting was expanded to include the following local government areas: City of Monash, City of Greater Dandenong, City of Casey, Cardinia Shire, City of Kingston, City of Glen Eira, City of Bayside, City of Frankston, Knox City, and the Shire of Mornington Peninsula. The final study setting area consists of approximately 1.392 million Victorians, or 24% of Victoria’s population [35]. It includes affluent areas and semi-rural growth corridors as well as the most socially and financially disadvantaged area in metropolitan Melbourne (the City of Greater Dandenong), with 56% of its residents born overseas and high numbers of refugees [36]. It also includes areas with disproportionately high numbers of retirees and older Australians in the Shire of Mornington Peninsula.

A list of the final participating study sites can be obtained from the contact author, but only if sites consent to this.

Eligibility criteria

GP criteria

To be eligible, practice sites are required to meet the accreditation standards for quality of care and risk management set by the Royal Australian College of General Practitioners [37]. GP eligibility criteria are: having worked at their current practice for at least 12 months, with minimum 2.5 days per week at the study site and a majority of that work in generalist primary care. Enrolment involves committing to participate in the recovery-oriented training intervention, to identify eligible patients and to distribute study invitation letters and surveys to potential patient participants.

Patient criteria

Base inclusion criteria for patient participants are:

  • aged 18 years and over

  • aged less than 75 years of age

  • proficient in English

  • able to provide informed consent

  • patients of a participating GP, that is, consult with the GP in 50% of their visits to the practice or is identified by the GP as a patient.

In addition to the base criteria, of the ten patients to be recruited at each of the 3 time periods (see Fig. 1), seven are to have at least one of the following:

  • a recent mental health plan made, or

  • a review of a mental health plan, or

  • prescribed any class of antidepressant medication on a continuing basis as treatment for a mental illness; and

three are to have at least one of the following:

  • a diagnosis of psychosis (e.g., schizophrenia, schizoaffective disorder, bipolar disorder), or

  • been prescribed antipsychotic medication in the previous 6 months

Individuals in prison are excluded.


The intervention consisted of training modules and optional ongoing participation in monthly webinars with participating GPs. Before development of the training materials, individual semi-structured interviews with GPs explored potential barriers to recovery-oriented training within primary care settings and this information was used to assist in design. The training intervention was designed in alignment with the modular training pathway of the General Practice Mental Health Standards Collaboration Mental Health Skills Training (MHST) requirements [38] and received accreditation for this as well as accreditation through the two relevant Australian GP Professional Colleges. Patients of GPs who have completed MHST can claim higher rebates for some mental health care activities Medicare.

The two modules of the recovery-oriented training intervention delivered to participating GPs are: 1. Core Module (3.5 h) with learning objectives including increased skill in recognizing and assessing common mental illnesses within a recovery-oriented framework; a greater working knowledge of the Better Access initiative and mental health treatment planning with a focus on operationalizing recovery-oriented practice in general practice; enhanced understanding of the perspective of consumers and carers in the provision of mental healthcare; and increased knowledge of the local mental healthcare services and resources available to GPs; and 2. Clinical Enhancement Module (CEM; 4 h), which provides the opportunity to apply knowledge gained in the Core Module within the specific context of Schizophrenia. Learning objectives of the CEM include: development of skills in the detection and assessment of Schizophrenia; an ability to apply the principles of recovery-oriented practice to treatment planning and monitoring; the ability to develop recovery-focused mental health treatment plans; and an applied understanding of review processes and relapse prevention strategies for mental illness within a recovery-oriented practice framework. MHST criteria are met by participation in both modules, along with preparatory and reinforcing exercises. For GPs who already have MHST accreditation, the option is open to attend the CEM only, and one round of the training is delivered as something of a hybrid between the two but with full retention of the recovery-oriented practice content. In the latter two situations, GPs retain access to professional development points through their college but the MHST requirements do not need to be met.

In addition to GPs, other clinic/practice staff such as practice nurses, non-participating GPs, and administration staff are also encouraged to attend training. Training is delivered by mental health clinicians, including experienced trainers from the study team, with co-delivery by consumer trainers. An experienced family/carer worker also participates, providing a minimum of one hour of content over the total of eight hours of training. The intervention draws heavily on the REFOCUS program [15] and the General Practitioner Mental Health Treatment Plan-Recovery (GPMHTP-R). The latter was developed in the Better Mental Health Treatment Plans Project, which was conducted by the Southern Synergy group with the support of a Commonwealth Government funding grant. The recovery-oriented training intervention includes training in the use of the GPMHTP-R, and also features locally developed materials. The intervention was developed following consultation and discussion with a group of local GPs (n = 7).

Each training participant is provided with resource material consisting of background reading and the Primary Care PULSAR Manual, which explains how to incorporate recovery-oriented practice into regular practice. The Project Team is responsible for developing, evaluating, and making these resources readily available. A schedule of training is provided to trainers, along with standardized training packs (including videos) to ensure the provision of standardized content. Trainers are asked to keep a record of training and to record any deviations from the schedule and use of materials.

PALS: PULSAR active learning sessions

GPs and other professionals who have received the PULSAR training are invited to participate in monthly one-hour online sessions called “PALS (PULSAR Active Learning Sessions)” with consultant specialist psychiatrists to review, reflect and share their experiences in the implementation of recovery-oriented practice. These sessions provide an interactive learning environment for supporting practice-based implementation of learning from the PULSAR resources and training package.

Intervention modifications and delays

Any delays in implementing the training during the protocol timelines are noted and the actual date of training recorded. The analysis plan has been expanded to include all available data with the actual training date in an attempt to maximize the informational value from the collected data (see Statistical analysis section).


To account for potential reduction of the intervention ‘dosage’ within each cluster, dosage is measured as the percentage of GPs who undergo the recovery-oriented practice training. This variable is based on the percentage of cluster staff who complete the training and remain in the cluster. For example, if two GPs at a cluster receive the training intervention at Step 1 and both remained employed at the practice/clinic/health center, this dosage variable would be 100% at Step 2.

Access to study intervention at study closure

At the end of the study period GPs and their staff will continue to have access to the training resources provided, except for PALS. It is anticipated that the resources developed will inform ongoing initiatives by the local Primary Health Networks.


Clusters were randomized to receive the intervention at either Step 1 or Step 2 (see Fig. 1) using a minimization procedure [39]. To ensure that each step period had a balance of cluster types, stratified randomization was applied using four types of organizational variations (see the cluster level stratification variables in Table 2). The study statistician performed the stratified randomization through December 2014 to June 2015, and sites were notified shortly afterwards.

Breaking the cluster intervention code

Breaking the cluster intervention code necessarily occurred after randomization (see above) so that training could be organized at the clusters. However, only key people involved with the organization and delivery of the training were informed of the intervention/training schedule.

Quantitative methods


The original study protocol (as documented in the Australian New Zealand Clinical Trials Registry, or ANZCTR) was developed over a period of 18 months and involved consultation with the Chief Investigators (CIs) and local stakeholders, and included representation from primary care, community care, and patients. The planning team included primary care academics familiar with the known barriers associated with recruitment of GPs into research projects [4042].

The research team originally planned to deliver the recovery-oriented practice training intervention to one participating GP per cluster site, and to evaluate the process using a study design of a cRCT conducted over four years, utilizing a two-step stepped-wedge design. This design was developed to combine the rigor of a cluster randomized trial with the pragmatic approach of the stepped-wedge design to implement the intervention at all sites. However, initial challenges with GP engagement and patient recruitment resulted in necessary modifications to the original project protocol. These are outlined below. All adaptations to the study protocol were considered by, and required the approval of, the appropriate Module Committee governing the relevant aspect of the project (see Study leadership section). Half-yearly project reports to the funding body also advised of changes made or intended. In each case these were then considered by the agent of the funding body. Queries as raised in some cases were responded to by the investigators prior to acceptance of the report by the funding body agents. Adaptations to the original protocol as reported here were thereby given endorsement from the funding body. Despite the adaptations described below, the basic two-step stepped-wedge cRCT design remains unchanged (see Fig. 1) and the trial will conclude within the planned four-year timeframe.

Adaptations to design

Delays in recruitment of GPs and patients and in delivery of the intervention required some modification to the design in order to be able to deliver the project outcomes within the allowed project duration. Cluster definition was broadened to include one or more GPs from each site in an effort to boost recruitment. As well, study steps were reduced from 12 months to 9 months. Based on the Kirkpatrick training evaluation model [43], the 9-month step period is an acceptable time point for the measurement of change. It allows 3 months for consolidation and a 6-month period for patients to be exposed to GPs implementing recovery-oriented practice.


All outcome measures are listed in Table 1, and the study survey is available upon request to the contact author. The primary outcome measure, the 22-item Questionnaire about the Process of Recovery (QPR) [44], enables an examination of the primary research question by collecting information from patients of participating GPs about personal recovery. This patient-rated outcome measure was chosen for the study end-point as personal recovery is experienced by an individual rather than assessed by an expert.

Table 1 Primary, secondary and other outcome measures in PULSAR Primary Care

The secondary outcome measures include the importance of services in recovery questionnaire (INSPIRE) [45], which assesses the recovery orientation of the GP service from the perspective of the patient, and two measures of mental health and wellbeing, the Warwick-Edinburgh Mental Well-being Scale (WEMWBS) [46], and the Kessler Psychological Distress Scale (K10) [47].

Additional measures are: Participant Demographic Record and the General-practice-Users Perceived-need Inventory (GUPI) [48]. As well, a section in the survey is designed to gain an understanding of any financial burden of illness as well as the participants’ use of health services. These health economic and service utilization questions, which contain both study-specific items and questions adapted from the 2007 Australian National Survey of Mental Health and Wellbeing [49], include information on the participants’ job status, income, consultations with health professionals, medications being taken, admissions to hospital, and any time away from usual roles because of illness. Participants are advised that if they cannot answer a question precisely, then to provide a best estimate and that there are no ‘right’ or ‘wrong’ answers. Finally, where specific consent is provided, information related to diagnosis, mental health status and current medication is also collected from GP medical records.

Sample size targets

The study has two sample groups: GPs and patients of these GPs. Sample size calculations were done using Stata statistical software stepped-wedge: for clusters defined at the level of the general practice site, the primary outcome measure is the QPR, with a published mean of 46 and standard deviation of 16, power of 0.80, significance level set at 0.05, intraclass correlation coefficient (ICC) within practice sites of 0.05, the number of steps (2), and patient data collected cross-sectionally at three time points (baseline, Step 1, and Step 2).

The original sample target was a minimum of 20 different sites (clusters) with one GP per site; each GP was required to recruit a minimum of 30 patients during the four-year project (10 patients at baseline, 10 at Step 1, and 10 at Step 2), resulting in a minimum of 600 patient participants overall.

Adaptations to sample size targets

Challenges in engaging GPs and recruiting patients were encountered at the beginning of the project (see earlier and also below). As a result, the protocol was adapted so that more than one GP from each cluster can participate. Each participating GP is still expected to recruit a minimum of 30 patients over the duration of the trial; therefore, if there are two participating GPs at a cluster, each will recruit 30 patients, resulting in 60 patients for that cluster. This efficiency of scale appears to be working well in some clusters. The required cluster number was also reduced from 20 to 10 plus. These sample numbers will be sufficient to detect a medium to medium-large effect in the primary outcome.

GP recruitment

Over 325 general practice and community health centers in the Monash Health catchment area were identified by the research team, using the National Health Services Directory and the telephone directory. The team also accessed a comprehensive database of GPs who have been involved in GP research in the south-east region of Melbourne held by the Department of General Practice at Monash University. Existing contacts within relevant health care organizations that coordinate service delivery related to primary care, such as the Medicare Locals were utilized. As noted earlier, since this phase of the study, these organizations have been replaced by Primary Health Networks.

An Invitation to Participate Letter was sent to each potentially eligible GP site, which outlined the study and requested participation. The letter was addressed to the lead physician(s), owner(s), or manager at each address and was signed by the project Principal Investigator (PI) and two CIs (both general practitioners). Letters of invitation to the community health centers also included a letter of support from Monash Health, who governs the centers. A PULSAR team member made follow-up telephone calls four-to-seven days after letters were posted to explore interest and, if appropriate, request a meeting with senior site staff to explain the study further. If a GP decided to participate, she or he was required to sign the GP Participant Information and Consent Form.

Simultaneously, the study was advertised and GP participation sought using various newsletters and websites.

Adaptations to GP recruitment and retention

In addition to allowing more than one GP per cluster site to be recruited and expanding the geographical area of participating practices, as mentioned above, increased efforts were made to engage the Medicare Locals in promoting the study. This led to a sufficient number of GPs (n = 30) being recruited.

Patient recruitment

Participating GPs are responsible for organizing patient recruitment at their site. Recruitment is coordinated by the participating GP independently of the researchers, to maintain the privacy of patient participants and minimize the research-related administrative burden placed on practice site staff. Assistance is offered by the PULSAR researchers in training relevant practice staff in identifying potentially eligible patients.

GPs were requested initially to identify around 50 eligible patients from their practice site. Each cluster site was originally provided with a base remuneration of $200 for committing resources to help offset administration costs involved in recruitment. An additional $25 is received by the site for the successful recruitment of each eligible patient, and an additional $25 is sent to patients who return the survey in the provided return envelopes to the researchers.

Strategies to assist participating GPs are informed by the known impediments to recruitment in general practice [4042]. The primary mode of patient recruitment is through survey packs that are sent in mail outs or handed out by each practice site. The packs contain an invitation to participate letter, a participant information and consent form, the study instruments, and two reply-paid envelopes. These envelopes are addressed to the PULSAR team rather than to patients’ clinics in order to reduce possible breaches of confidentiality; consenting participants return their data directly to the research team, who then securely enter (using password-protected files and computers) and store (using lockable filing cabinets for hard copies) all information. GPs and/or practice managers are required to oversee the mailing or handing out of survey packs.

Recruitment strategies were designed from the start to be flexibly employed according to the needs of each practice. A number of secondary strategies were also developed to promote patient response, including offering on-site availability of researchers, advertising via practice sites, and GPs handing out information flyers to eligible patients.

Practice billing and clinical software is used to identify those patients treated by the participating GP in the previous three months who have had a mental health plan drawn up or a review of a mental health plan conducted and/or have been prescribed any class of antidepressant medication on a continuing basis (for at least a month) as treatment for a mental illness. Billing and clinical software is also used to identify patients of the participating GP who have been prescribed antipsychotic medication in the previous six months or have had a diagnosis of psychosis. The participating GP is required to screen all lists of potential participants, with reference to their clinical notes if necessary, to confirm clinical status and eligibility for the study. It is emphasized that this step is important in the identification of patients with a diagnosis of psychosis as anti-psychotic medications are also prescribed for people without such a diagnosis.

Challenges with patient recruitment and retention

Initial patient recruitment by the cluster sites generated mixed results that varied from one cluster site completing baseline recruitment within one week to others not reaching their target after six months. Some clusters reported difficulties in identifying potentially eligible patients and a reluctance to mail to patients in case doing so jeopardized the practice/patient relationship. Other reasons for some cluster sites not distributing the survey to patients were: confusion about patient eligibility; believing the study was over or mistakenly thinking that the cluster had withdrawn from it; a lack of interest from patients in completing the surveys; and the required increase in administrative burden. Reasons given for being unable to identify eligible patients included resource pressures or having high numbers of patients that were either under the age of 18 or from culturally and linguistically diverse (CALD) backgrounds with low English proficiency.

Adaptations to patient recruitment

In addition to site visits, other strategies to assist clusters include an increase in baseline remuneration to $500, identification by the Project Manager of key personnel for liaison, and reductions of the administrative burden by the provision of relevant, clinic-tailored advice. A sub-study has been established to examine aspects of recovery-oriented practice in GP clinics with high CALD patient populations.

Procedures to minimize bias

The study GPs could not be blinded to the intervention allocation as the intervention involved GP training. Therefore, the following procedures to minimize other sources of bias have been adopted:

  1. 1.

    Wherever possible, the research assistants, administration staff, and data entry staff are blinded to the intervention allocation.

  2. 2.

    Patients enrolled: procedures undertaken to minimize contamination include that participating patients are not advised by the research team if their GPs has received the intervention training.

  3. 3.

    Control clusters: patients in the control clusters continue to receive treatment as usual. As the study has a stepped-wedge design, all sites will receive the intervention by Step 2. Stepped-wedge designs are often preferred for such community-based pragmatic trials as they can minimize contamination of control clusters as staff and clinicians in all sites know that the study intervention is eventually coming [3234].

  4. 4.

    Recruitment: capacity to give informed consent is presumed for the majority of people and provisions are made to ensure those most in need are given the opportunity to participate. For example, the recruitment sampling method outlined above allows for maximum numbers of patients to be identified who then have a choice in deciding whether or not they would like to participate in research. This is done to assist in avoiding possible sampling bias and gives flexibility for patients to respond based on possible fluctuations in mental health. Ethics research indicates that people with serious mental illness are able to provide informed consent to participate in research, especially if particular efforts are made to recruit them [50]. Research also indicates that, for people with mental health issues, participating in research can lead to positive reactions such as a sense of enjoyment and empowerment, and is beneficial for improved service delivery [51, 52]. It is presumed that those who do not wish to participate, or those who are unable to understand the study goals and procedures, will not return the consent forms and completed measures.

  5. 5.

    Allocation: all randomization was carried out by the statistician in the research team following the appropriate procedures set out earlier.

  6. 6.

    In analysis: wherever possible, the data collectors and data entry team are blinded to allocation status and do not have access to information about the allocation of clusters. Study data are entered into a study database that does not contain information about intervention status.

Statistical analyses

The recovery-oriented practice training intervention for primary care will be evaluated at the patient level by examining the surveys of adult patients who consult the participating GPs for mental health issues. The main evaluation plan is to examine the cross-sectional surveys returned by patients during three data collection periods: baseline, Step 1, and Step 2 (see Fig. 1).

The planned data collection schedule depends on the training intervention being delivered at each cluster in the scheduled month (see Fig. 1). As training delays in some clusters are expected, the following two main analysis approaches are planned to compare the post-intervention and control (pre-intervention) periods:

  1. 1.

    Study planned dataset, which will use data collected from patients during the planned six-month periods centered on the midpoints of February 2015, August 2015 and November 2015 (see Fig. 1). Other data collected outside of these timespans are considered a protocol violation and are excluded from this dataset.

  2. 2.

    Per-protocol all data dataset, which will use all available data time-stamped from date of intervention received at the cluster, as outlined in Table 3.

Main analysis plan

Descriptive statistics will be used to summarize the characteristics of the patient-level variables and the GP clinics (clusters), see Table 2. Cluster-level variables are those used in the stratified randomization, which are four types of organizational variations, plus the intervention status of the cluster and the time since (or before) the intervention. The intraclass correlation coefficient (ICC) will be calculated and reported.

Table 2 Individual and cluster-level variables available for multivariable analysis

The primary analysis will examine the patient-level QPR scores (continuous data) at baseline, nine months (Step 1), and 18 months (Step 2), using a linear mixed-effects model. The model will include intervention status and time as fixed effects and clusters and patients as random effects. Both univariate and multivariable models will be developed based on baseline patient and cluster-level variables considered statistically significant (p < 0.10) or clinically important (e.g., age, sex), see Table 2, and included in the model as fixed. This will include the intervention dosage variable described earlier. Model fit will be examined by comparing AIC values.

Secondary analyses will examine the patient-level data of the INSPIRE, WEMWBS, and K10. Similar to the primary analysis, we will use linear mixed-effects models to compare the intervention and control periods (pre-intervention) for continuous outcomes and generalized linear mixed-effects models for binary outcomes.

Estimated intervention effects will be reported as the mean outcome difference for continuous outcomes and Odds Ratio for binary outcomes between intervention and control periods, assuming a constant treatment effect over time. This can be described as a meta-analysis approach as (in the case of continuous data) the mean change in each cluster will be standardized by using the variance of the outcome measure within that cluster. The estimated intervention effects will be reported with 95% Confidence Intervals and p values. Analysis will be conducted using Stata V.14, StataCorp. Stata Statistical Software: Release 14. College Station, TX: StataCorp LP, 2015.

Sensitivity analyses

A missing data analysis will investigate any patterns of missingness. For each primary and secondary outcome component with missing data, multiple imputation using multivariate regression with factors of age, gender, time, and intervention status will produce 100 estimates. Sensitivity analyses will be performed using this multiple imputation to account for missing data and then re-running the analyses.

Economic evaluation

Overall, costs associated with each participant will follow well established health economic principles [53], and cover direct medical costs of illness, plus the labor market effects of illness. Direct medical costs are to be calculated for prescription medications and hospital and health service contacts. Labor market productivity losses will be imputed using the human capital approach by multiplying reported days off work due to mental illness with an individual’s estimated salary using instrumentation devised by this team for a previous health economic evaluation [54]. Only using days off work due to illness to capture labor market costs captures an important aspect of the cost of illness; however, it is noted that the estimates obtained will be conservative and the true cost will be higher than what we obtain because of other effects of illness such as higher rates of non-participation in employment, or underemployment.

Data collection and management

Table 3 outlines the data collected at each time point for participating GPs and patients. Patient participant data is being collected by the returned paper surveys or online surveys at three time points: baseline (months 4–6), Step 1 (months 16–18) and Step 2 (months 28–30), as displayed in Fig. 1. Both modes of survey completion are provided to offer flexible participation options. The estimated completion time for each round of surveys is approximately 45 min.

Table 3 Schedule of enrolment, interventions, and assessments

The online survey option collects participant-entered data directly from Qualtrics online survey software. Qualtrics securely collects participant responses within a secure back-end spreadsheet that is only accessible by the researchers via password protected files.

If patient participants consent to allowing the researchers to access their routinely collected medical records, additional information will be extracted from these files with the assistance of the participating GP/s in each cluster site, including information about diagnosis and mental health status and medication details relating to mental health issues.

All study data is stored in re-identifiable format, from which identifiers have been removed and replaced by a code. Re-identification is necessary in order to link survey data with routinely collected data for patient participants who consent to the release of this information. Any personal information such as participant names remain confidential and all information is stored in password protected files and folders on password protected computers located at the core PULSAR administration site. These can only be accessed by the research staff. The study data will be stored for a minimum of 7 years, after this time it may be confidentially destroyed. It may be possible that future research such as a meta-analysis will use the participant data in non-identifiable format.

Qualitative methods

The embedded qualitative study was designed to investigate the contextual enablers and challenges to implementing recovery-oriented practice in participating primary care settings through two stages of qualitative data collection. First, an interview schedule based on the Promoting Action on Research Implementation in Health Services (PARIHS) framework [55, 56] was used with a small number of GPs to identify potential barriers or challenges to delivering recovery-oriented practice training for GPs. This approach enabled the intervention sites’ readiness for recovery-oriented practice to be gauged, and the optimization of the intervention package and its implementation.

Second, semi-structured interview schedules, informed by literature on the implementation of recovery-oriented practice and consumer and service provider expertise within the PULSAR Qualitative Research Steering Group, were developed to investigate GP and patient views and experiences of recovery-oriented practice within primary care settings. These interviews are conducted face-to-face or by telephone at three and nine months following the GP training. Interviews with GPs explore their understanding of recovery-oriented practice and experiences and challenges encountered in implementing a recovery-oriented framework at practice level. Subsequently, participating GPs are invited to reflect on the de-identified interview themes, and on facilitators and barriers to implementing recovery-oriented practice in primary care settings, in an interview or focus group discussion. Similarly, face-to-face or telephone interviews conducted with patient participants focus on their views and experiences of recovery-oriented practices as used by GPs, with a subsequent opportunity to reflect on the de-identified interview themes and on supports for their recovery within and beyond services.

Sample size is determined sequentially by qualitative sampling processes to ensure diverse perspectives are sought, and to maximize the richness of data obtained, for which we anticipate at least 10 GP participants and 10 patient participants will be interviewed.

All qualitative data are audio-recorded (subject to participant consent) or documented in handwritten notes, then transcribed for coding and thematic analyses, so as to identify thematic similarities and differences within and across participant groups. All transcribed data are de-identified and along with all other PULSAR data are stored in password-protected files within the restricted access electronic files of the PULSAR site.

Process evaluation

The Project Team developed a fixed schedule of content for the training sessions for GPs and actively referred to this schedule during delivery. Following the guidelines provided by Moore, Audrey, Barker, Bond, Bonell, Hardeman et al. the process evaluation will focus on collecting data in the following four areas:

  1. 1.

    Description of the intervention and project assumptions (Module 1: Adaptation)

  2. 2.

    The implementation context (Module 2: Implementation)

  3. 3.

    Mechanisms of impact (Module 3: Research)

  4. 4.

    The outcomes of the study (Module 4: Dissemination).

This evaluation will yield data on the process of implementing a complex intervention design associated with the stepped-wedge approach such as sampling responses, effect sizes, and organizational and contextual factors that impact the study processes and outcomes. Qualitative and quantitative analysis provide evidence of the influence of contextual and implementation factors on outcomes. Together, the data will be integrated to identify barriers and enablers for achieving expected outcomes from the intervention and contribute explanatory detail with respect to main findings.


A project implementation group (which functions as a data monitoring committee) meets weekly, monitoring progress towards targets set by the module task groups, reporting and escalating issues to those task groups as necessary. These meetings are attended by the principle investigator, project manager and other core study staff. In these weekly meetings, protocols prepared or amended by the module task groups are operationalized to provide guidance to the study research team. Feedback is also collated on intervention implementation, maintenance, and monitoring of the overall conduct of the trial; this information is readily relayed to the relevant module task groups and principle investigator, thereby enabling timely assessment and intervention if necessary.

Leadership structure

The management and advisory structure of the PULSAR project adopts a module based approach. The trial is overseen by a project steering group (Chair: Professor Graham Meadows) and four modules guide and support the development, implementation and evaluation of the project. The module task groups and chairs are: Adaptation module (Chair: Christine Thornton & Graham Meadows); Implementation module (Chair: Penny Weller); Research module (Chair: Lisa Brophy); and Dissemination module (Chair: Vrinda Edan). The Lived Experience Advisory Panel (LEAP) is a consumer and family/carer advisory group that guided the development of the REFOCUS-PULSAR materials and training intervention and provides ongoing recommendations on the implementation of the trial.

Specification of safety parameters

No plans were made for a premature stopping of the trial. Participant (GP and patients) safety was classified as low risk, apart from any breaches to patient confidentiality, which were classified as moderate risk.

Safety oversight

Detailed project protocols have been developed addressing the management of participant distress, suicidal ideation or intent, threat to harm others, and disclosure of previously undisclosed criminal acts.


A stepped-wedge cluster randomized controlled trial in the general practice and community health center primary care settings is an innovative and novel approach to test a training intervention for general practitioners (GPs) in recovery-oriented practice. This intervention will enable GPs, and their staff, who provide primary care to people with mental illness in this study to be introduced to recovery-oriented practice and to recruit the people who consult them to be actively engaged in evaluating the effectiveness of the intervention through a mixed methods study design. This research design has both advantages and limitations [57], and the challenges encountered are providing important insights into methods to maximize the recruitment and retention of GPs in a large scale and complex study, as well as how to engage the patients of GPs who are presenting with mental ill health. If findings are positive from the work described, the intervention has the potential to extend recovery-oriented practice to GPs and thus reduce the current barriers to GPs and their patients gaining access to this system-wide transformation.

Dissemination policy


A multi-level approach to knowledge transfer towards influencing practice will be used. The plan will include: publication of a training manual and information leaflets; submission for publication in peer-reviewed literature of findings from each component of the project; presentations at international and national scientific and practice-focused conferences; web-site development for making project materials readily accessible to other interested parties; local dissemination through our partner organizations in Victoria’s mental health treatment and support sectors; and direct presentations to policy makers to ensure the findings are well understood and appreciated where key decisions are being taken.


In relation to copyright issues in dissemination of findings, PI Meadows and CI Slade have agreed to highly accessible publication to maximize dissemination. Specifically, there is no plan to commercialize outputs of this work and so put barriers in the way of use by others. It has been the practice of the multiple research teams involved in the PULSAR proposal actively to seek to make materials widely available without cost, and to place barriers in the way of others commercializing such work. For example, the London REFOCUS team have disseminated the REFOCUS intervention in free-to-access booklets and through open access journal articles. The dissemination plan will make the findings widely and readily available along with source training materials.



Australian New Zealand Clinical Trials Registry


Chief Investigator


Cluster randomized control trial


General practitioner mental health treatment plan-recovery


General-practiceusers' perceived-need inventory


Human research ethics committee


Intraclass correlation coefficient


Lived experience advisory panel


Mental health skills training


National health and medical research council


Promoting action on research implementation in health services


Principal investigator


Principles unite local services assisting recovery


Questionnaire about the process of recovery


Stepped-wedge cluster randomized controlled trial


Warwick-edinburgh mental well-being scale


  1. Burgess PM, Pirkis JE, Slade TN, Johnston AK, Meadows GN, Gunn JM. Service use for mental health problems: findings from the 2007 National Survey of Mental Health and Wellbeing. Aust N Z J Psychiatry. 2009;43(7):615–23. doi:10.1080/00048670902970858.

    Article  PubMed  Google Scholar 

  2. Britt H, Miller GC, Henderson J, Bayram C, Harrison C, Valenti L, et al. General practice activity in Australia 2014–15. General practice series no. 38. Sydney: Sydney University Press; 2015.

    Google Scholar 

  3. Britt H, Miller GC, Charles J, Henderson J, Bayram C, Pan Y et al. General practice activity in Australia 2008–09. Cat. no. GEP 25. Australian Institute of Health and Welfare. 2009. Accessed 27 Jul 2016.

  4. Australian Institute of Health and Welfare. Expenditure on mental health services. 2015. Available at Accessed 27 Jul 2016.

  5. Stone L, Blashki G. The role of GPs in mental health care. In: Meadows G, Farhall JF, Fossey EM, Grigg M, McDermott FM, Singh B, editors. Mental health in Australia: collaborative community practice. 3rd ed. South Melbourne, Victoria: Oxford University Press; 2012. p. 275–9.

    Google Scholar 

  6. Australian Institute of Health and Welfare. Mental health-related services provided by general practitioners. 2015. Available at Accessed 25 Jul 2016.

  7. Victoria State Government. About Victoria’s mental health services. 2015. Accessed Sept 27 2016.

  8. Department of Health and Aging. National Mental Health Report 2013: tracking progress of mental health reform in Australia, 1993–2011. Canberra: Commonwealth of Australia; 2013.

    Google Scholar 

  9. Shepherd G, Boardman J, Slade M. Making recovery a reality. London: Sainsbury Centre for Mental Health; 2008.

    Google Scholar 

  10. Australian Health Ministers Advisory Council. A national framework for recovery-oriented mental health services: guide for practitioners and providers. Canberra: Commonwealth Government; 2013.

    Google Scholar 

  11. Anthony WA. Recovery from mental illness: the guiding vision of the mental health service system in the 1990s. Psychosoc Rehabil J. 1993;16(4):521–38.

    Google Scholar 

  12. Leamy M, Bird V, Le Boutillier C, Williams J, Slade M. Conceptual framework for personal recovery in mental health: systematic review and narrative synthesis. Bri J Psychiatry. 2011;199(6):445–52. doi:10.1192/bjp.bp.110.083733.

    Article  Google Scholar 

  13. Slade M, Amering M, Oades L. Recovery: an international perspective. Epidemiol Psichiatr Soc. 2008;17(2):128–37.

    Article  PubMed  Google Scholar 

  14. Slade M, Leamy M, Bacon F, Janosik M, Le Boutillier C, Williams J, et al. International differences in understanding recovery: systematic review. Epidemiol Psychiatr Sci. 2012;21(4):353–64.

    Article  CAS  PubMed  Google Scholar 

  15. Slade M, Bird V, Le Boutillier C, Farkas M, Grey B, Larsen J, et al. Development of the REFOCUS intervention to increase mental health team support for personal recovery. Br J Psychiatry. 2015;207(6):544–50.

    Article  PubMed  Google Scholar 

  16. Slade M, Bird V, Clarke E, Le Boutillier C, McCrone P, Macpherson R et al. Supporting recovery in patients with psychosis through care by community-based adult mental health teams (REFOCUS): a multisite, cluster, randomised, controlled trial. Lancet Psychiatry. 2015;2(6):503–14. doi:10.1016/S2215-0366(15)00086-3.

  17. Gerrand V, Singh B, Nagel T, Ash D, Turnbull C, Meadows G, et al. Mental health services in Australia. In: Meadows G, Farhall JF, Fossey EM, Grigg M, McDermott FM, Singh B, editors. Mental health in Australia: collaborative community practice. 3rd ed. South Melbourne, Victoria: Oxford University Press; 2012. p. 69–117.

    Google Scholar 

  18. Ash D, Bland R, Brown P, Browne MO, Edwards R, Emmerson B, et al. Mental health services in the Australian states and territories. In: Meadows G, Farhall JF, Fossey EM, Grigg M, McDermott FM, Singh B, editors. Mental health in Australia: collaborative community practice. 3rd ed. South Melbourne, Victoria: Oxford University Press; 2012. p. 118–54.

    Google Scholar 

  19. Meadows GN, Enticott J, Castle D. Psychiatry and primary care. In: Bloch S, Green S, Janca S, Mitchell P, Robertson M, editors. Foundations of Clinical Psychiatry. 4th ed.: Melbourne University Publishing; in press

  20. Bird V, Leamy M, Le Boutillier C, Williams J, Slade M. REFOCUS: Promoting recovery in community mental health services. London: Rethink. 2011.

    Google Scholar 

  21. Edan V, Meadows G, Brophy L, Weller P, Thornton C. PULSAR research: making mental health services more recovery-orientated. New Paradigm: Aust J Psychosoc Rehabil. Elsternwick. Victoria: Psychiatric Disability Services of Victoria; 2015. p. 46–8.

    Google Scholar 

  22. Elwyn G, Frosch D, Thomson R, Joseph-Williams N, Lloyd A, Kinnersley P, et al. Shared decision making: a model for clinical practice. J Gen Intern Med. 2012;27(10):1361–7.

    Article  PubMed  PubMed Central  Google Scholar 

  23. Stubblefield C, Mutha S. Provider-patient roles in chronic disease management. J Allied Health. 2002;31(2):87–92.

    PubMed  Google Scholar 

  24. Brody DS. The patient’s role in clinical decision-making. Ann Intern Med. 1980;93(5):718–22.

    Article  CAS  PubMed  Google Scholar 

  25. Stewart M, Brown JB, Donner A, McWhinney IR, Oates J, Weston WW, et al. The impact of patient-centered care on outcomes. J Fam Pract. 2000;49(9):796–804.

    CAS  PubMed  Google Scholar 

  26. Chan A-W, Tetzlaff JM, Gøtzsche PC, Altman DG, Mann H, Berlin JA, et al. SPIRIT 2013 explanation and elaboration: guidance for protocols of clinical trials. BMJ. 2013;346:e7586.

    Article  PubMed  PubMed Central  Google Scholar 

  27. Creswell JW. A concise introduction to mixed methods research. Thousand Oaks: Sage Publications; 2014.

    Google Scholar 

  28. Creswell JW, Clark VLP. Designing and conducting mixed methods research. Thousand Oaks: Sage Publications; 2007.

    Google Scholar 

  29. Moore GF, Audrey S, Barker M, Bond L, Bonell C, Hardeman W, et al. Process evaluation of complex interventions: Medical Research Council guidance. BMJ. 2015;350:1–7. doi:10.1136/bmj.h1258.

    Article  Google Scholar 

  30. Wallace G, Bird V, Leamy M, Bacon F, Le Boutillier C, Janosik M, et al. Service user experiences of REFOCUS: a process evaluation of a pro-recovery complex intervention. Soc Psychiatry Psychiatr Epidemiol. 2016;51(9):1275–84. doi:10.1007/s00127-016-1257-9.

    Article  PubMed  Google Scholar 

  31. Bhanbhro S, Gee M, Cook S, Marston L, Lean M, Killaspy H. Recovery-based staff training intervention within mental health rehabilitation units: a two-stage analysis using realistic evaluation principles and framework approach. BMC Psychiatry. 2016;16(1):292. doi:10.1186/s12888-016-0999-y.

    Article  PubMed  PubMed Central  Google Scholar 

  32. Brown CA, Lilford RJ. The stepped wedge trial design: a systematic review. BMC Med Res Methodol. 2006;6(1):54.

    Article  PubMed  PubMed Central  Google Scholar 

  33. De Allegri M, Pokhrel S, Becher H, Dong H, Mansmann U, Kouyaté B, et al. Step-wedge cluster-randomised community-based trials: an application to the study of the impact of community health insurance. Health Res Policy Syst. 2008;6(1):10.

    Article  PubMed  PubMed Central  Google Scholar 

  34. Mdege ND, Man M-S, Taylor CA, Torgerson DJ. Systematic review of stepped wedge cluster randomized trials shows that design is particularly used to evaluate interventions during routine implementation. J Clin Epidemiol. 2011;64(9):936–48.

    Article  PubMed  Google Scholar 

  35. Health M. Monash Health Fast Facts 2014–15. 2015. Accessed 27 Jul 2016.

    Google Scholar 

  36. Australian Bureau of Statistics. National Regional Profile: Greater Dandenong City. 2010. Accessed 26 Jul 2016.

  37. The Royal Australian College of General Practitioners. Standards for general practices (4th ed.). 2015. Accessed 26 Jun 2015.

  38. General Practice Mental Health Standards Collaboration. Mental health education standards 2014–2016: a handbook for training providers. Melbourne: The Royal Australian College of General Practitioners; 2013.

    Google Scholar 

  39. Scott NW, McPherson GC, Ramsay CR, Campbell MK. The method of minimization for allocation to clinical trials: a review. Control Clin Trials. 2002;23(6):662–74.

    Article  PubMed  Google Scholar 

  40. White D, Hind D. Projection of participant recruitment to primary care research: a qualitative study. Trials. 2015;16(1):473. doi:10.1186/s13063-015-1002-9.

    Article  PubMed  PubMed Central  Google Scholar 

  41. Foster JM, Sawyer SM, Smith L, Reddel HK, Usherwood T. Barriers and facilitators to patient recruitment to a cluster randomized controlled trial in primary care: lessons for future trials. BMC Med Res Methodol. 2015;15(1):18–27. doi:10.1186/s12874-015-0012-3.

    Article  PubMed  PubMed Central  Google Scholar 

  42. Horspool MJ, Julious SA, Mooney C, May R, Sully B, Smithson WH. Preventing and lessening exacerbations of asthma in school-aged children associated with a new term (PLEASANT): recruiting primary care research sites–the PLEASANT experience. NPJ Prim Care Respir Med. 2015;25:15066. doi:10.1038/npjpcrm.2015.66.

    Article  PubMed  PubMed Central  Google Scholar 

  43. Kirkpatrick JD, Kirkpatrick WK. Kirkpatrick’s four levels of training evaluation. Alexandria: Association for Talent Development; 2016.

    Google Scholar 

  44. Neil ST, Kilbride M, Pitt L, Nothard S, Welford M, Sellwood W, et al. The questionnaire about the process of recovery (QPR): a measurement tool developed in collaboration with service users. Psychosis: Psychological, Social and Integrative Approaches. 2009;1(2):145–55. doi:10.1080/17522430902913450.

    Article  Google Scholar 

  45. Williams J, Leamy M, Bird V, Le Boutillier C, Norton S, Pesola F, et al. Development and evaluation of the INSPIRE measure of staff support for personal recovery. Soc Psychiatry Psychiatr Epidemiol. 2015;50(5):777–86.

    Article  PubMed  Google Scholar 

  46. Tennant R, Hiller L, Fishwick R, Platt S, Joseph S, Weich S, et al. The Warwick-Edinburgh Mental Well-being Scale (WEMWBS): development and UK validation. Health Qual Life Outcomes. 2007;5(1):63–76. doi:10.1186/1477-7525-5-63.

    Article  PubMed  PubMed Central  Google Scholar 

  47. Kessler RC, Andrews G, Colpe LJ, Hiripi E, Mroczek DK, Normand SL, et al. Short screening scales to monitor population prevalences and trends in non-specific psychological distress. Psychol Med. 2002;32(6):959–76.

    Article  CAS  PubMed  Google Scholar 

  48. McNab C, Meadows G. The General-practice Users’ Perceived-need Inventory (‘GUPI’): a brief general practice tool to assist in bringing mental healthcare needs to professional attention. Prim Care Mental Health. 2005;3(2):93–101.

    Google Scholar 

  49. Australian Bureau of Statistics. National Survey of Mental Health and Wellbeing of Adults: User’s Guide (4327.0). 2008. Available at Accessed 24 Jul 2014.

  50. Roberts LW. Evidence-based ethics and informed consent in mental illness research. Arch Gen Psychiatry. 2000;57(6):540–2. doi:10.1001/archpsyc.57.6.540.

    Article  CAS  PubMed  Google Scholar 

  51. Browne G, Hemsley M. Consumer participation in mental health in Australia: what progress is being made? Australas Psychiatry. 2008;16(6):446–9. doi:10.1080/10398560802357063.

    Article  PubMed  Google Scholar 

  52. Jorm AF, Kelly CM, Morgan AJ. Participant distress in psychiatric research: a systematic review. Psychol Med. 2007;37(7):917–26. doi:10.1017/s0033291706009779.

    Article  PubMed  Google Scholar 

  53. Glied S, Smith PC. The Oxford handbook of health economics. New York: Oxford University Press; 2011.

    Book  Google Scholar 

  54. Shawyer F, Enticott JC, Özmen M, Inder B, Meadows GN. Mindfulness-based cognitive therapy for recurrent major depression: a ‘best buy’ for health care? Aust N Z J Psychiatry. 2016. doi:10.1177/0004867416642847

  55. Stetler CB, Damschroder LJ, Helfrich CD, Hagedorn HJ. A guide for applying a revised version of the PARIHS framework for implementation. Implement Sci. 2011;6(1):99.

    Article  PubMed  PubMed Central  Google Scholar 

  56. Rycroft-Malone J. The PARIHS framework - a framework for guiding the implementation of evidence-based practice. J Nurs Care Qual. 2004;19(4):297–304.

    Article  PubMed  Google Scholar 

  57. Palmer VJ, Chondros P, Piper D, Callander R, Weavell W, Godbee K, et al. The CORE study protocol: a stepped wedge cluster randomised controlled trial to test a co-design technique to optimise psychosocial recovery outcomes for people affected by mental illness in the community mental health setting. BMJ Open. 2015;5(3):e006688. doi:10.1136/bmjopen-2014-006688.

    Article  PubMed  PubMed Central  Google Scholar 

Download references


We acknowledge Prof Mike Slade, Dr Mary Leamy and Dr Vicki Bird and other members of the REFOCUS team. We also acknowledge, Mr Umit Agis, A/Prof Jakqui Barnfield, Dr Joanne Brooker, Ms Kimberly Buck, Prof David Clarke, Dr Margaret Grigg, Mr John Julian, Dr Ritsuko Kakuma. A/Prof Iraklis Harry Minas, A/Prof Brendan Murphy, Dr Mehmet Ozmen and Ms Christine Thornton.


This article presents the protocol for independent research conducted within the PULSAR project, a project funded by the Victorian Government, Department of Health’s Mental Illness Research Fund (MIRF project #18). On 1 January 2015, this department became the Department of Health & Human Services, Victoria, Australia. The views, analyses, interpretations, and conclusions expressed in the article are those of the authors, not of the Victorian Department of Health & Human Services.

Availability of data and materials

Only investigators and approved researchers added by ethics approval will have access to the final trial data set.

Authors’ contributions

All authors participated in the design of the trial and intervention. LB chaired the research module task-group and provided oversight to development of all elements of the design. PW chaired the implementation module which oversaw the delivery of the training intervention. JE and GM led development of key elements of the design and analysis approach with GR and DM. GM, DM, VE, SV, PW and GR developed the specific primary care training intervention and associated resources. FS, GR, DM, JE, BI, SV and GM developed the primary care instrumentation and fieldwork trial protocols. Key contributions regarding process evaluation and qualitative research were made by EF and EWE. All authors read and approved the final manuscript.

Competing interests

The authors declare that they have no competing interests.

Consent for publication

Not applicable.

Ethics approval and consent to participate

Research ethics approval

The project was approved by Human Research Ethics Committees at Monash Health (project approval number 14146B and 14427B) and Monash University (project approval number CF14/2422 – 2014001304 and CF15/266 - 2015000120). Original protocol approvals were granted on 12 August 2014 before project commencement and extend to 12 August 2019. All protocol amendments were submitted to, and approved by, both committees using the appropriate ethic amendment forms.

Research audit

An independent audit of related PULSAR research procedures, focusing on the correct use and storage of participant data, was undertaken by Human Research Ethics Committees at Monash Health in the third year and no anomalies were identified.

Patient consent to participate

Signed consent is obtained from all participants in both the quantitative and qualitative studies. Any information gained in connection with this research project that can identify individuals will remain confidential. All information will be stored in password protected files and folders on password protected computers. These can only be accessed by the research staff and possible future related projects. If any information can identify individuals, such as participant contact details or the signed consent form, will not be stored near or directly linked to the data provided.

Quantitative study

Individuals who receive survey packs via a mail out or directly from their GP return their signed consent forms via mail. It is assumed that those who do not wish to participate, or who are unable to understand the goals and procedures of the study will not return consent forms and completed measures. Completed surveys that are returned without signed consent forms are excluded from analysis.

The patient Participant Information and Consent Form and Invitation to Participate letter explain that potential participants are required to return the signed consent form and complete the surveys. The letter also explains that: a) potential participants can, if they prefer, complete the surveys online; b) their GP has not released any identifying information to the researchers and it is the individual’s choice to disclose such information for the purposes of research participation as indicated by completing and returning the consent form and survey; c) in addition to completing the survey (Level 1 consent), they may provide additional permission to enable the researchers to access routinely collected (and relevant) clinical data to be used in the project (Level 2 consent) and/or consent to being contacted for future research related to the PULSAR project by ticking the appropriate sections on the consent form. Potential participants are instructed to return the survey and consent form in the envelopes provided in the packs.

Qualitative study

Participants in the quantitative study who indicate that they are willing to be contacted for future research related to the PULSAR project are mailed a separate pack containing an Invitation to Participate in the Nested PULSAR Qualitative Study letter, a Participant Information and Consent Form, a contact information sheet and a reply paid envelope. The letter explains that participation in the qualitative study involves one to two interviews exploring recent experiences of the mental health care received since the PULSAR training was provided to their GP or mental health service staff. The Participant Information and Consent Form explains that interested individuals may either a) tick the appropriate box on the consent form, complete the contact information sheet and return both to Southern Synergy using the reply paid envelope; or b) telephone the research team to learn more about the study. The study recognizes that consenting is an ongoing process during research participation. The researchers re-confirm participants’ agreement to participate in interviews and/or focus groups prior to appointments and reiterate what is involved on each occasion.

GP consent to participate

For both the quantitative and qualitative studies, eligible GPs are mailed Invitation to Participate letters outlining what their participation in the study or studies would involve and are invited to sign and return a GP Participant Information and Consent Form. As with the patient data collected during this study, all information gained from GPs in connection with this research project will remain confidential using the methods mentioned earlier.

Discontinuing participants

All participants in both the quantitative and qualitative studies, including GPs and patients, are advised at the beginning of participation in the study and in the consent form that they can discontinue participation at any time. Data already collected will still be used in the analysis unless the discontinuing participant specifically requests that it be removed.

Author information

Authors and Affiliations


Corresponding author

Correspondence to Graham Meadows.

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (, which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver ( applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Enticott, J.C., Shawyer, F., Brophy, L. et al. The PULSAR primary care protocol: a stepped-wedge cluster randomized controlled trial to test a training intervention for general practitioners in recovery-oriented practice to optimize personal recovery in adult patients. BMC Psychiatry 16, 451 (2016).

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: