Skip to main content
  • Study protocol
  • Open access
  • Published:

A minimum evaluation protocol and stepped-wedge cluster randomized trial of ACCESS Open Minds, a large Canadian youth mental health services transformation project



Many Canadian adolescents and young adults with mental health problems face delayed detection, long waiting lists, poorly accessible services, care of inconsistent quality and abrupt or absent inter-service transitions. To address these issues, ACCESS Open Minds, a multi-stakeholder network, is implementing and systematically evaluating a transformation of mental health services for youth aged 11 to 25 at 14 sites across Canada. The transformation plan has five key foci: early identification, rapid access, appropriate care, the elimination of age-based transitions between services, and the engagement of youth and families.


The ACCESS Open Minds Research Protocol has multiple components including a minimum evaluation protocol and a stepped-wedge cluster randomized trial, that are detailed in this paper. Additional components include qualitative methods and cost-effectiveness analyses. The services transformation is being evaluated at all sites via a minimum evaluation protocol. Six sites are participating in the stepped-wedge trial whereby the intervention (a service transformation along the key foci) was rolled out in three waves, each commencing six months apart. Two sites, one high-population and one low-population, were randomly assigned to each of the three waves, i.e., randomization was stratified by population size. Our primary hypotheses pertain to increased referral numbers, and reduced wait times to initial assessment and to the commencement of appropriate care. Secondary hypotheses pertain to simplified pathways to care; improved clinical, functional and subjective outcomes; and increased satisfaction among youth and families. Quantitative measures addressing these hypotheses are being used to determine the effectiveness of the intervention.


Data from our overall research strategy will help test the effectiveness of the ACCESS Open Minds transformation, refine it further, and inform its scale-up. The process by which our research strategy was developed has implications for the practice of research itself in that it highlights the need to actively engage all stakeholder groups and address unique considerations in designing evaluations of complex healthcare interventions in multiple, diverse contexts. Our approach will generate both concrete evidence and nuanced insights, including about the challenges of conducting research in real-world settings. More such innovative approaches are needed to advance youth mental health services research.

Trial registration number, ISRCTN23349893 (Retrospectively registered: 16/02/2017).

Peer Review reports


Mental disorders usually begin before the age of 25 [1] and can persist [2], sometimes in changing forms [3]. They often disrupt the pursuit of the typical social and educational developmental milestones of youth and impose much suffering and societal cost [2, 4, 5]. Mental illness is the leading contributor to disability-adjusted life years among people aged 0–24 years in high-income countries [6]. In Canada, suicide accounts for 20–25% of deaths among 15- to 24-year-olds [7]. Canada’s Indigenous youth suffer exceptionally high rates of mental health problems, substance abuse, and suicide [8, 9].

Adolescence and young adulthood represent a critical window for early detection and intervention because that is when most mental illnesses arise [1] and are presumably most amenable to treatments that can improve long-term outcomes and reduce disease burden [10, 11]. Moreover, longer durations of untreated mental illnesses are generally associated with worse clinical and functional outcomes and greater personal and familial suffering [12,13,14]. Yet, many youth with mental health problems remain untreated or face delayed detection [15].

In Canada, where 18.5% of youth aged 15 to 24 are affected by mental and substance use disorders [16], a 2012 survey found that only 49.7% of youth with mental health problems and 25.8% of those with substance use disorders had sought professional help in the preceding year [17]. Our systematic review [18] found that help-seeking youth’s pathways to mental healthcare were often complex, stressful/negative (e.g., involving emergency rooms or police) and delay-prone. Across much of Canada, long waiting lists significantly delay evaluation and treatment [19] and youth are also excluded from care based on not meeting specific diagnostic criteria [20]. Canadian youth mental healthcare is also marred by improper transitions between child-adolescent and adult services; a paucity of evidence-informed interventions; fragmented, inefficient and siloed system components [21, 22]; and little involvement of patients and their families in service design and planning, despite growing acknowledgement of the value of such involvement [23,24,25,26]. Consequently, services fail to engage and meet the needs of youth and their families.

Addressing these issues is the remit of ACCESS Open Minds, a research network established under the Strategy for Patient-Oriented Research (SPOR) [24] of the Canadian Institutes of Health Research (CIHR). This five-year project draws on the experiences of Australian, Irish and British youth mental healthcare improvement initiatives and previous work on early intervention in psychosis [27,28,29] Partnering with multiple stakeholder groups (youth and families with lived experience, service providers, researchers, and decision-makers), ACCESS Open Minds developed, and is implementing and evaluating a transformation of services for youth (aged 11 to 25) with all types and severities of mental health difficulties at 14 sites across Canada [30].

ACCESS Open Minds is implementing strategies to increase youth referrals and help-seeking; accelerate response times; provide timely access to appropriate interventions as per established benchmarks; eliminate age-based transitions between services; and engage youth and families. The transformation also involves creating youth-friendly, stigma-reducing, community-based spaces that allow walk-in access to integrated services and supports. It is hypothesized that this transformation will improve the reach, timeliness, appropriateness, quality, acceptability, clinical and functional outcomes, and subjective experience of mental health services. The ultimate goal is to create a sustainable and scalable model for timelier, friendlier, and more effective youth mental healthcare.


This study evaluates the effectiveness of the ACCESS Open Minds model using a multi-method design that includes a minimum evaluation protocol, a pragmatic stepped-wedge randomized trial, a qualitative strategy, and a cost-effectiveness evaluation. This paper focuses on the minimum evaluation protocol and the stepped-wedge trial. The qualitative and economic evaluations will be described in separate publications.

Given the breadth of issues addressed, strategies deployed and populations targeted, we ask the following main questions in this study:

  • How and to what extent does ACCESS Open Minds work to identify youth in need and improve their access to high-quality mental healthcare?

  • Among which youth groups and in which contexts is the transformation most and least beneficial?

These research questions are operationalized through well-defined primary and secondary objectives and related hypotheses.

Primary objectives

To determine if the ACCESS Open Minds model:

  1. 1.

    Increases early case identification

    Hypothesis 1: There will be a significant increase in the number of youth seeking help at, or being referred to, study sites over the course of the project.

  2. 2.

    Reduces systemic delay in responding to referrals/help-seeking

    Hypothesis 2: The proportion of youth being offered an initial assessment by a trained clinician within 72 h of referral/help-seeking will increase and be sustained at or above benchmark levels (once attained) over the course of the project.

  3. 3.

    Reduces treatment delay (i.e., the delay between the initial evaluation and the commencement of appropriate treatment)

    Hypothesis 3: Over the course of the project, a higher proportion of youth seeking services at sites will be offered appropriate care/interventions within 30 days (the Canadian Psychiatric Association’s benchmark) [31]. For urgent cases, the guidelines recommend commencing treatment in less than 30 days. This will be examined separately for youth with and without serious mental illnesses.

Secondary objectives

To determine if the ACCESS Open Minds model:

  1. 4.

    Simplifies pathways to mental healthcare

    Hypothesis 4: Over the course of the project, youth will make fewer help-seeking contacts in the previous 12 months before accessing services at project sites.

  2. 5.

    Improves outcomes for youth served at project sites

    Hypothesis 5: The clinical, social-vocational and subjective functioning of youth will significantly improve over the course of their follow-up. An additional objective is to determine which outcome domains are most impacted by treatment. The lengths of follow-up will vary depending on youth’s needs, severity of illness, etc.

  3. 6.

    Satisfies youth and their families

    Hypothesis 6: At least 75% of service users and their families/carers will be satisfied with services and service providers at all completion time-points.



ACCESS Open Minds’ 14 sites represent pan-Canadian variations in geography, culture, resources, and population density [30] (see Table 1). Together, they serve youth in urban, rural, and remote communities; non-Indigenous and Indigenous youth; Anglophone and Francophone youth; and groups with particular vulnerabilities (immigrants, refugees, ethnic minorities, state-protected youth, homeless youth, and first-year university students). Ten sites cover specified catchment areas, strengthening the representativeness of the youth populations they serve.

Table 1 ACCESS Open Minds Site Descriptions (A list of study sites with addresses can be obtained from the trial registry)

Six sites are in Indigenous communities, where populations tend to be young [32]. In some Indigenous communities, youth are reported to have very high rates of suicidality, addiction, violence, school drop-out, unemployment, and involvement with justice and youth protection systems [33]. It has been proposed that these problems stem from and are compounded by the cultural fragmentation and inter-generational trauma inflicted by brutal colonial and government policies [34]. The situation is exacerbated by varying combinations of geographic isolation (e.g., two sites are only accessible by air and boat); poverty; the inadequacy of health and psychosocial services and associated funding; and service providers’ unfamiliarity with local histories, languages and contexts [35].

Study intervention

At each site, the ACCESS Open Minds “intervention” transforms services [21, 22] to provide:

  1. 1.

    Early case identification through targeted outreach, community awareness, etc., so that more youth self-refer or are referred sooner (see online early identification guide) [36].

  2. 2.

    Rapid, engaging access, offering an initial evaluation within 72 h in a non-emergency, community-based environment. To this end, a trained “ACCESS Clinician” is deployed to conduct initial evaluations; involve family members in assessment; and link youth with services tailored to their needs and preferences. Rapidity is also ensured through the plurality of portals of access, including direct walk-in; the elimination of referral or administrative requirements; and the use, where appropriate, of technologies like social media, helplines, etc.

  3. 3.

    Appropriate care in the form of evidence-informed, illness-appropriate interventions offered within 30 days of initial evaluation (per Canadian Psychiatric Association benchmarks) [31]. Because youth can present with non-specific, overlapping symptoms, treatment planning is guided by youths’ self-reported distress and social-vocational functioning and clinicians’ impressions of problem severity, rather than specific diagnoses. Provided in youth-friendly, non-stigmatizing, and recovery-oriented environments, care is focused on youth’s own goals. Where appropriate interventions are unavailable, site staff connect youth to external services/specialists.

  4. 4.

    Continuity of care, to ensure that youth receive appropriate care for as long as needed. This is achieved by fostering collaboration between services, stakeholders, sectors, and disciplines to eliminate jarring age-based transitions and to reduce or smoothen transitions between other services youth need, e.g., from primary to specialized care.

  5. 5.

    Youth and family/carers engagement, to ensure their active involvement, in keeping with the SPOR vision of valuing lived experience. This entails including youth and families in network- and site-level service design, oversight, and hiring committees; seeking their inputs in designing youth spaces; offering individualized menus of interventions, flexible appointment times and, when possible, choices of treatment venues; and training clinicians in youth-friendly, strengths-affirming approaches.

At all sites, transformation began with a community mapping [37,38,39] exercise that identified, linked up and, where possible, co-located all youth-focused resources in the community (see online guide) [40]. Additional capacity was integrated through hiring an ACCESS Clinician and other requisite staff, training, and external partnerships (e.g., with the national helpline, Kids Help Phone). Additional details about the model and its operationalization appear in our prior open-access publication [30].

Although a dynamic process, service transformation is deemed to commence at each site with: (a) the finalization of a site-specific service transformation plan (on the basis of which a contract is signed and funds are allocated) with clear activities outlined to achieve each service objective; (b) the deployment of the ACCESS Clinician; and (c) the training of key site staff in ACCESS Open Minds principles and evaluation protocols.

Core strategies common to all sites’ transformation plans include deploying an ACCESS Clinician; responding to help-seeking/referrals within 72 h; and creating a youth-friendly physical space. Relevant evidence and local conditions inform additional site-specific strategies. For instance, twice-yearly early identification activities may target all youth at sparsely populated Indigenous sites, but only potential referral sources (e.g., schools) at more populous urban sites. The relevance, acceptance and ownership of each site’s service transformation plan are assured through broad stakeholder engagement.

Study design

The impact of ACCESS Open Minds is being evaluated using a multi-method design that includes a minimum evaluation protocol at all sites, and a stepped-wedge cluster randomized trial at six sites.

Minimum evaluation protocol: Our minimum evaluation protocol — developed collaboratively in consultation with youth, families, researchers, and clinicians — is designed to collect data that address our primary and secondary hypotheses (see Table 2). These include measures/indicators of: service contexts (e.g., numbers of youth in the catchment area); service users (e.g., number of males accessing services); service provision/processes (e.g., portals of entry available, types of early identification strategies deployed, etc.); and outcomes/impacts (e.g., time between help-seeking and initial evaluation, satisfaction with services, reduction in distress, etc.).

Table 2 Summary of assessment tools selected for the study

Measures/indicators were selected based on the following priorities:

  1. 1.

    Sustainability, to increase the likelihood of the protocol’s continued use after the research project, consistent with the ideal of measurement-based care [41,42,43].

  2. 2.

    Wide applicability, so that measures/indicators address research objectives; inform treatment and quality improvement; and enjoy buy-in, especially from sites that had little prior research or evaluation experience. To ensure wide applicability, we chose measures/indicators targeting multiple informants, including youth, families/carers, and clinicians/service providers.

  3. 3.

    Appropriateness, to choose measures shown to have adequate psychometric properties in varied Canadian youth populations. Some measures were chosen because they had been used in population-wide surveys (Canadian Community Health Survey-Mental Health [44] and Aboriginal People’s Survey [45]. This will allow the comparison of data from our sites with corresponding population-level data. When several options were available, shorter, more user-friendly measures were chosen to enhance feasibility and acceptability.

Data are collected throughout the study period, and over-time comparisons will be conducted at the network and individual site levels. At some sites, we will also conduct pre-post comparisons.

Stepped-wedge trial: Six sites were selected to participate in a randomized stepped-wedge trial [46, 47] wherein the intervention (a transformed service) was sequentially rolled out in three clusters/waves, starting six months apart. Sites were rank-ordered based on population, and the first three ranked sites were grouped as high and the last three ranked sites were grouped as low population. Thus stratifying by population size, two sites, one high-population and one low-population, were randomly assigned to each wave (see Fig. 1). At an event attended by representatives from all six sites, the randomization assignment was conducted by drawing chits from separate urns containing names of large and small sites (see Fig. 2).

Fig. 1
figure 1

Flowchart of trial using SPIRIT flowchart recommendations

Fig. 2
figure 2

Configuration of waves after randomization in April 2015

All stakeholders deemed the stepped-wedge design preferable on ethical grounds because it deployed the transformation at all sites, rather than having some sites serve solely as “controls.” For their capacity to generate high-impact evidence (like traditional RCTs), stepped-wedge trials are increasingly used to evaluate complex service-based and public health interventions [48]. They involve the random and sequential crossover of waves from the control condition to the intervention until all waves are exposed.

The lack of concealment of the allocation of the intervention can introduce selection biases to stepped-wedge trials [49,50,51]. It may also be difficult to separate the effects of the intervention from changes that simply happen over time. However, this can be countered statistically [52]. Such trials are prone to logistical constraints, particularly with complex, multi-component interventions [49,50,51]. For instance, the intervention may commence or attain adequacy at a site later than when its cluster was randomized to begin the intervention due to external events.

Per the stepped-wedge design, data collection continues throughout the study at all sites, so that each cluster contributes observations under both control and intervention conditions. All six sites thus act as controls whilst also undergoing transformation. Refer to Additional file 1: SPIRIT Checklist [53, 54], which lists where specific trial details can be found in this paper.

Project management and participation plan

Our earlier publication [30] outlines the project’s governance structure and processes. Research is guided by a Research Advisory Group comprising representatives from all sites; network-level youth and family/carers council representatives; and network-level researchers. Overall guidance is provided by a Steering Committee of four external researchers who, together, have expertise in youth mental health, primary healthcare, Indigenous issues, and biostatistics.

At each site, evaluations are conducted by local staff, who were trained by the Montréal-based central office team, tasked with supporting sites and operationalizing the project. Site staff also receive booster training and ongoing support. At regular sessions where all sites participate, multiple raters rate the same scenarios independently on two clinician-administered measures (see Table 2), discuss ratings, and establish consensus guided by experts. These sessions help raters calibrate themselves in relation to a scale’s anchors and ensure that scales are reliably rated across sites.

Videos [55,56,57] were created to explain the project, the role of evaluations, and informed consent to site staff and youth. Videos featuring actors were also developed to demonstrate best practices for introducing evaluations and informed consent to youth. The project’s national youth council created and shared engaging, youth-friendly posters describing the project and its research/evaluation strategy.

Participant recruitment, consent and withdrawal

The sites will recruit youth for a period of 3–3.5 years until September 2020. The first participant was enrolled in the study in July 2016. Participants include youth and, where possible, their families/carers.

While details vary across sites, the general recruitment procedure is for the ACCESS Clinician or another staff member to identify new referrals to a research assistant/staff who explains the project to them and seeks written informed consent (see Additional file 2 for sample consent form). Consenting youth are enrolled into the study.

The protocol recommends approaching youth for written consent at the time of intake. However, recruitment can be deferred if youth are in crisis situations or too busy. For minors below the age of 14 or youth with diminished capacity, consent is sought from a parent/legal representative along with assent from the young person. Provincial/institutional regulations vary in terms of policy around the need for consent from a parent/legal representative in the case of youth between the ages of 14 and 18 (see Declaration about “Ethics approval and consent to participate”).

Consent is also sought from family members for their participation in family-focused assessments. Signed forms are filed securely in locked cabinets and copies are given to participants.

Consent is voluntary and can be withdrawn at any time, with no impact on access to services or the quality of care. Consent simply allows data from measures completed by youth and from their health administrative records to be recorded and analyzed for the research project.

Following youth feedback, a consent form (see Additional file 2) was created that, while modeled on standard ethics guidelines, minimized length and legalese. It describes the services offered, details the research/evaluation component, and clarifies that consenting to research is not a prerequisite to receiving services. It thus respects the project’s philosophy of integrating care and evaluations. Across sites, the wording or structure of consent forms varies to respect local ethics procedures and cultural considerations. Sites have also developed recruitment strategies suited to their context and youth population. Some sites offer honoraria/compensation to youth for time and travel costs incurred in completing assessments, and others often meet youth outside, in the community or a café, to complete assessments, etc.

Inclusion and exclusion criteria: To recruit as representative a sample as possible of all youth served, inclusion criteria are broad and exclusion criteria minimized. For enrollment into research/evaluation, youth must be aged 11 to 25 years and be seeking help for a mental health and/or substance-related problem. Youth who received services at a site before its transformation are eligible if they have not received any services for six months or longer. This was done to ensure that the study of outcomes is restricted to those receiving transformed services. Individuals younger than 11 years or older than 25 years; with a diagnosed intellectual disability; with a history of organic brain damage; and/or deemed unable to consent are not eligible. For family members/carers to be included in the research, they must be connected to a youth receiving services at a site and provide informed consent.

Outcome measures

Outcome measures are summarized in Table 2 and their psychometric properties in Additional file 3. Measures that were available only in English were translated into French with forward and back translation and a focus on conceptual equivalence [58, 59]. All purpose-designed forms are available in English and French. Measures of clinical, functional and subjective outcomes are administered at baseline and repeated for those who receive further treatment.

Sample description: Socio-demographic variables like sex, age, housing situation, ethnicity, and immigration background are obtained from a self-report form, clinical interviews, or records. Such data are crucial given the known social determinants of mental healthcare access and outcomes.

Where possible, items were picked to allow population-level comparisons, e.g., a Canadian census indicator of visible minority status [60] to help determine whether sites serve as many minority youth as the ethnic composition of their catchment populations would suggest they should.

A clinical profile of service recipients is created using a purpose-designed checklist of presenting concerns, the Global Assessment of Individual Needs-Short Screener [61], and the Columbia-Suicide Severity Rating Scale screener [62], among other tools..

Service-level outcomes: Forms were purpose-designed to record information about sites’ performance on key indicators pertaining to our three main hypotheses (increase in number of youth assessed, evaluation within 72 h, and adherence to the 30-day benchmark).

Data from these forms will also help identify barriers to meeting project benchmarks (e.g., delay due to lack of an on-site psychiatrist) and site-level quality improvement (e.g., information on referral sources will help tailor early case identification activities). Secondary questions can also be asked, e.g., about whether self-referrals increase over the course of the project. Staff will record on these forms up to three services needed by youth following the initial evaluation and services received by them 30 days later, which will serve as our measure of “appropriate” care.

Items taken from the Canadian Community Health Survey-Mental Health (CCHS-MH) 2012 survey [44] are used to determine whether youth are first-time help-seekers and the number and types of help-seeking contacts they make before entering a site (Hypothesis 4).

Clinical outcomes: To gauge the extent of clinical improvements, both self-rated and observer-rated measures are used. This includes the Kessler Psychological Distress Scale (K10) [63], a self-report measure of distress, and two self-rated single-item measures of physical and mental health [64, 65]. These measures are quick, easy to complete, validated, and used widely in clinical and population surveys, including in Canada [66]. The K10 is also used in Australian youth mental health hubs [67, 68].

Clinicians or trained research staff rate the severity of youth mental health presentations using the single-item Clinical Global Impression (CGI) scale [69]. Because youth present with highly diverse problems, anchor descriptions from the trans-diagnostic version [70] of the CGI were adopted. At all time points after the initial evaluation, staff also rate improvement in symptoms using the single-item CGI-Improvement. The CGI was chosen over more elaborate measures to increase clinician buy-in and rates of completion.

Functional outcomes: To examine if transformed services improve functioning, the Social and Occupational Functioning Scale (SOFAS) completed by clinicians or research assistants will be used. The SOFAS [71] is rated on a 1-to-100 continuum, without considering the severity of symptoms. In addition, the work/school status of youth is also being recorded.

Subjective outcomes and well-being: Three measures assess youth’s subjective progress and well-being following intervention — outcomes that were prioritized by youth members of the network.

The minimally burdensome Outcome Rating Scale assesses four dimensions of life functioning that indicate the success of therapeutic interventions [72]: individual well-being (measuring personal or symptom distress); interpersonal well-being (measuring functioning in intimate relationships); social role (measuring satisfaction with work/school and non-familial relationships); and overall well-being. At select sites, youth also complete the World Health Organization Quality of Life (WHOQOL-BREF) [73], a quick measure of perceived quality of life.

Given the larger goal of evaluating impacts for all youth across settings, most measures in our protocol are nomothetic. However, being a patient-oriented research project, it was thought important to include an idiographic measure inspired by the Goals Based Outcome (GBO) [74]. It allows youth to rate their progress toward three self-identified, personally salient goals during their follow-up. Family members/carers are also invited to choose and rate progress towards three goals in relation to their loved ones’ services. Pooled data on goal attainment will help evaluate the effectiveness of services, as has been done in the United Kingdom with the GBO [75,76,77].

Service experiences: To assess the extent to which ACCESS Open Minds transforms cultures of care, the youth and family versions of the Ontario Perceptions of Care (OPOC) [78] (a validated tool developed in Canada) will be used to assess satisfaction with the accessibility, quality and other aspects of services. Therapeutic alliance will be assessed using the Session Rating Scale (SRS) and youth’s perceptions of empowerment and engagement using the Youth Efficacy/Empowerment Scale [79]. These tools complementarily assess alliance with individual care providers (SRS) [80] and overall satisfaction with a service (OPOC).

Data collection procedures

At all sites, data collection by trained clinical and research staff begins with referral and continues throughout follow-up, the length of which varies according to the needs of youth. Research assistants need 1 to 1.5 h to complete initial assessments and about 30 min for follow-up assessments. They have access to staff for help with any emergent crises.

In this study, “data” refers to data obtained directly from consenting individuals (e.g., self-reports) and non-identifiable data from pertinent health administrative records obtained with institutional approval. A secure electronic data capture and data management system makes the research/evaluation protocol accessible online from anywhere. Project staff, youth, and families can use the system to complete measures and forms on any device (desktop, tablet, or phone). Self-administered measures and forms can be emailed to youth/families for future completion on a user-friendly interface. The entire system — measures, forms, and interface — is bilingual (French and English). An integrated scheduling function facilitates follow-ups at time points as per the protocol. In case of connectivity issues, data can be collected offline and uploaded later. People who prefer completing measures/forms on paper can do so. Whether collected on paper or digitally, all data are entered into the electronic system.

Once data are entered, customized reports can be easily extracted to inform care in real time; e.g., clinical progress reports can support intervention decisions and goal attainment reports can help demonstrate improvement to youth. Dashboards within the software can be customized to display frequently sought information.

We expect to generate valuable insights into customizing and deploying electronic data capture and management systems in diverse real-world service and research settings. We will also seek to understand whether such a user-friendly system promotes measurement-based mental healthcare, which, despite its well-acknowledged value [41,42,43], is rare or inconsistent in Canada [81].

Periodic audits will be conducted at all sites to verify adherence to ethical and recruitment procedures, and to verify accuracy of data being collected. This audit will be undertaken by research team members based in a central coordinating office in Montréal. Central office staff also will regularly review electronic data to flag double data, data values not being in range, etc.

Ethical considerations

Having obtained ethics approval from pertinent institutional bodies (see Additional file 4), the study is following appropriate regulations and standards regarding data privacy, confidentiality, storage, and security (see Declaration about “Ethics approval and consent to participate”). A checklist is being used to record adverse events (e.g., death by suicide, accidents, etc.), which are communicated to site and central institution ethics committees and to the project steering committee.

This study is being conducted according to Good Clinical Practice [82]. Appropriate procedures include informed consent from individuals, and institutional approval for access to pertinent non-identifiable health administrative records. Data are stored securely, and confidentiality of data will be ensured. A steering committee has been set up to oversee the project and provide inputs to the team of investigators.

ACCESS Open Minds acknowledges Canada’s problematic history of academic and administrative research on Indigenous peoples (outlined in the 1996 Report of the Royal Commission on Aboriginal Peoples) [83]. We follow the Ownership, Control, Access, and Possession (OCAP™) principles [84] and other precepts of the guidelines for Research Involving the First Nations, Inuit and Métis Peoples of Canada from the Tri-Council Policy Statement on Ethical Conduct for Research Involving Humans [85].

Within the ACCESS Open Minds project, OCAP principles are being implemented at all levels. Specifically, community leadership and the ACCESS Open Minds Indigenous Council have played integral roles in partnering with Indigenous communities, guiding conversations about the implementation of the research and evaluation protocol, and bringing together important voices and perspectives from the community. The OCAP principles are underscored in all research and partnership agreements held between the ACCESS Open Minds project leads and Indigenous communities. Most importantly, key decisions– from youth space design to how data will be analyzed and interpreted -- are being guided by Indigenous site teams, including the site team leader, Elders, youth, and other interested community members. OCAP also addresses the issues of privacy, intellectual property, data custody, and secondary use of data.

In ACCESS Open Minds, following Tri-council guidelines for research involving Indigenous peoples [85] is being seen not merely as a requirement, but as congruent with Indigenous self-determination and full-spectrum stakeholder engagement, both of which are critical in effecting lasting change in youth mental health outcomes in Indigenous communities. Finally, Indigenous communities participating in ACCESS Open Minds also respect the Jordan’s Principle [86] that guarantees equitable access to healthcare services for Indigenous children, regardless of the healthcare jurisdiction in which they live or access services.

Sample size estimation

To arrive at the targeted sample size for catchment-area based sites (see Table 1), we used CCHS-MH-estimated percentages of the province's (a) youth who had a mental health/substance use disorder; (b) youth with mental disorders who sought help, and (c) youth with unmet needs for mental healthcare [87]. These were prorated to the site catchment’s youth population (i.e., number of youth aged 10 to 24 based on the 2016 Census [88]).

The estimate for unmet needs was added in furtherance of our primary objective of increasing case identification and thereby reducing untreated prevalence. Sites were to target reducing 20% of the unmet needs estimate per year in the project. Estimates for non-catchment area-based sites and for Indigenous community sites were arrived at in consultation with the sites.

The study protocol recommends that sites actively conduct outreach and other early identification activities; not turn away any youth; and approach all youth entering services for consent for research. At sites that have never participated in research, the percentage of youth who will provide consent cannot be estimated. It is therefore difficult to predict exactly how many youth will ultimately be included in our study. However, preliminary numbers from ongoing data collection suggest that 60% of youth approached will provide consent. It is estimated that the project will serve at least a total of 6029 youth. With a 60% consent rate, the estimated number of research participants from all 14 sites by the end of the project will be 3616. This should provide sufficient number or sample to answer the primary hypotheses.

Statistical plan

Every attempt will be made to collect outcome data from all participants, including those who do not complete treatment. Youth will be considered as having a serious mental illness if they attain the relevant scale-specific cut-off scores on the K-10, SOFAS and/or the CGI-severity scale.

Pre-post minimum evaluation protocol: For each site, general linear models with repeated measures will be used to analyze differences over time per site (for hypotheses pertaining to increased referrals, evaluations within 72 h, treatment within 30 days, number of prior help-seeking contacts, and satisfaction with services) and per youth (at baseline and at follow-up timepoints for the hypothesis pertaining to improvements in functioning). Generalized estimation equations will be employed for parameter estimation. Respective baseline values will be added to the model as a covariate. Additional analyses will include examining hypothesis 3 separately for youth with and without serious mental illnesses, examining inter-site heterogeneity, and identifying baseline predictors of response to treatment. Finally, outcomes will be examined separately for large urban, small urban/rural and Indigenous contexts.

Trial: Youth demographics and clinical data will be described using means and standard deviations and trial waves will be compared using t-tests or non-parametric tests for quantitative characteristics. The effect of the interventions on the three primary and all secondary outcomes will be quantified using mixed or multi-level models to integrate cluster/wave and time effect. The unidirectional crossover design of the stepped-wedge trial allows us to test the effect of the intervention at both within- and between-cluster levels. Mixed models allow stratification (e.g., by population sizes); inclusion of missing data with the principle of intention to treat; and adjustment for baseline differences.

Dissemination plan

Findings will be disseminated to various stakeholder groups via appropriate channels, including our project’s website (; user-friendly reports, charts, and slide shows for youth, families, sites, and decision-makers; and conference presentations and peer-reviewed journal publications for scientific audiences, including a publication outlining findings pertaining to our primary hypotheses. Dissemination of findings from Indigenous communities will respect OCAP™ principles [84]. Authorship guidelines for the project are available upon request and fall under the purview of the project’s national publications committee.


ACCESS Open Minds delivers and evaluates a large-scale innovation in youth mental health services that adds to, leverages and transforms extant services and resources. While critical additional human, training and material resources are required, the project does not create entirely parallel services distinct from Canada’s publicly funded healthcare system [89]. The project’s broad scope includes youth with widely varying mental healthcare needs from highly diverse geographic, cultural, economic and socio-demographic backgrounds [90,91,92,93,94,95,96]. An additional strength is that the ACCESS Open Minds model is informed by multiple relevant stakeholders, including youth, families/carers, researchers, clinicians, Indigenous communities, community organizations, and policy−/decision-makers.

Should its innovative, stakeholder-informed methods and strategies yield clear positive impacts within five years, ACCESS Open Minds can serve as a scalable, sustainable model for transformational change in the organization and delivery of youth mental healthcare in Canada and beyond.

Our multi-method research will show the extent to which the transformation increases and accelerates the early identification, assessment, and appropriate treatment of youth with mental health care needs; and improves pathways into care, satisfaction with services, and a range of mental health outcomes.

Our quantitative analysis will examine impacts over time at the network-wide and individual site levels. This, along with our planned qualitative analyses, will help us understand whether and why the transformation fades, intensifies, or differs across communities. Such insights will inform the applicability of the ACCESS Open Minds model across jurisdictions and will have significant policy implications.

Our carefully chosen metrics strengthen the internal validity of our future findings. Their external validity is enhanced by the diversity of our sites (many serving treated incidence samples) and populations; the investigation of multiple service variables; and the situating of our transformation in “real system, real resources” contexts.

Finally, the development and implementation of our research strategy offer valuable insights into: academic-stakeholder research partnerships; the integration of evaluations in settings that have limited experience with research/evaluations; and the building of research capacity among diverse stakeholders.


The potential of our stepped-wedge trial is limited because only six, very diverse sites were randomized to only three clusters. More, larger, and more similar clusters would have yielded a more robust trial [49,50,51]. Moreover, the dynamic, real-world nature of the transformation has posed practical challenges that impeded strict adherence to the timelines of our stepped-wedge protocol.

Two Indigenous sites, Ulukhaktok and Puvirnituq, were unable to follow the Minimum Evaluation protocol described in this paper because of cultural considerations, local constraints and preferences. They have also adopted a model that relies on local community workers to promote mental health literacy and wellness, and connect youth in need to supports, in lieu of hiring an “ACCESS Clinician”.

Across sites, the hesitancy of some inexperienced staff members to approach youth for consent could suppress sample sizes. The sheer diversity of local contexts and staff may result in reduced inter- and intra-site uniformity in recruitment and research assessment practices. This is being mitigated by creating a community of practice, conducting inter-rater reliability sessions, and providing training and support.

Our focus on evaluating the service transformation model in its entirety limits our ability to draw conclusions about the effectiveness of specific components. Our sample size and its distribution across 14 sites may preclude sophisticated analyses of the transformation’s differential impacts on different mental health problems. This will be mitigated by asking select questions about whether the model has different effects on presenting problems of differing gravity (mild-moderate versus serious).

Because the ACCESS Open Minds model involves the context-sensitive application of strategies to attain a set of five common objectives, the processes involved in service transformation and the extent to which objectives are attained can vary across sites. This is likely to influence the overall results. The actual processes of transformation at each site are being recorded and will be used to interpret results and to conduct additional explanatory analyses. Thus recording transformation processes will also provide rich, useful implementation data from diverse contexts that can inform similar initiatives in the future.

Finally, because data will be collected for different durations at different sites and for relatively modest lengths of time post-transformation, it will not be possible to assess the long-term site- or network-level impacts of service transformation.


Data from our research strategy will help assess, refine and scale up the ACCESS Open Minds model and generate concrete evidence to advance youth mental health services research. The collaborative development of our research strategy highlights the need to actively engage all stakeholders and address unique considerations in designing evaluations of complex healthcare interventions. Our participatory, integrated knowledge translation approach involved all key stakeholder groups in identifying and shaping research priorities, processes, methods, and measures. Such involvement, unprecedented in a project of this scale, can inspire other patient-oriented research undertakings. Our use of an electronic data capture and management system is significant given the marked underuse of such systems in Canadian mental health services and health services research [81, 97]. By deploying a common data management system and evaluation protocol in diverse settings, this project will also promote much-needed consensus on using common measures, indicators, and data collection strategies across youth mental health services in Canada. Finally, in executing and evaluating real, transformative actions, ACCESS Open Minds has the potential to catalyze practice- and policy-level change.



Canadian Community Health Survey-Mental Health


Clinical Global Impression


Goals Based Outcome


Kessler Psychological Distress Scale- ten-item version


Ownership, Control, Access, and Possession principles


Randomized controlled trial


Social and Occupational Functioning Scale


Standard Protocol Items: Recommendations for Interventional Trials


Strategy for Patient-Oriented Research


  1. Kessler RC, Berglund P, Demler O, Jin R, Merikangas KR, Walters EE. Lifetime prevalence and age-of-onset distributions of DSM-IV disorders in the National Comorbidity Survey Replication. Arch Gen Psychiatry. 2005;62(6):593–602.

    Article  PubMed  Google Scholar 

  2. Copeland WE, Wolke D, Shanahan L, Costello EJ. Adult functional outcomes of common childhood psychiatric problems: a prospective, longitudinal study. JAMA Psychiatry. 2015;72(9):892–9.

    Article  PubMed  PubMed Central  Google Scholar 

  3. Lee TY, Lee J, Kim M, Choe E, Kwon JS. Can we predict psychosis outside the clinical high-risk state? A systematic review of non-psychotic risk syndromes for mental disorders. Schizophr Bull. 2018;44(2):276–85.

    Article  PubMed  PubMed Central  Google Scholar 

  4. Bloom D, Cafiero E, Jane-Llopis E, Abrahams-Gessel S, Bloom LR, Fathima S, et al. The global economic burden of noncommunicable diseases. Geneva: World Economic Forum; 2011.

    Google Scholar 

  5. Ormel J, Oerlemans AM, Raven D, Laceulle OM, Hartman CA, Veenstra R, et al. Functional outcomes of child and adolescent mental disorders. Current disorder most important but past psychiatric history matters as well. Psychol Med. 2017;47(7):1271–82.

    Article  CAS  PubMed  Google Scholar 

  6. Erskine HE, Moffit TE, Copeland WE, Costello EJ, Ferrari AJ, Patton G, et al. A heavy burden on young minds: the global burden of mental and substance use disorders in children and youth. Psychol Med. 2015;45(7):1551–63.

    Article  CAS  PubMed  Google Scholar 

  7. Navaneelan T. Suicide rates: an overview. Statistics Canada: Canada; 2012.

    Google Scholar 

  8. Nelson SE, Wilson K. The mental health of indigenous peoples in Canada: a critical review of research. Soc Sci Med. 2017;176:93–112.

    Article  PubMed  Google Scholar 

  9. Canada H. Acting on what we know: preventing youth suicide in first nations. Health Canada: Ottawa; 2003.

    Google Scholar 

  10. McGorry PD, Hartmann JA, Spooner R, Nelson B. Beyond the “at risk mental state” concept: transitioning to transdiagnostic psychiatry. World Psychiatry. 2018;17(2):133–42.

    Article  PubMed  PubMed Central  Google Scholar 

  11. Mcgorry PD, Mei C. Early intervention in youth mental health: Progress and future directions. Evid Based Ment Health. 2018;21(4):182–4.

    Article  PubMed  Google Scholar 

  12. Wang PS, Berglund P, Olfson M, Pincus HA, Wells KB, Kessler RC. Failure and delay in initial treatment contact after first onset of mental disorders in the National Comorbidity Survey Replication. Arch Gen Psychiatry. 2005;62(6):603–13.

    Article  PubMed  Google Scholar 

  13. Dell’osso B, Altamura AC. Duration of untreated psychosis and duration of untreated illness: new vistas. CNS Spectr. 2010;15(4):238–46.

    Article  PubMed  Google Scholar 

  14. Penttila M, Jasskelainen E, Hirvonen N, Isohanni M, Miettuenen J. Duration of untreated psychosis as predictor of long-term outcome in schizophrenia: systematic review and meta-analysis. Br J Psychiatry. 2014;205(2):88–94.

    Article  PubMed  Google Scholar 

  15. Farmer EM, Burns BJ, Phillips SD, Angold A, Costello EJ. Pathways into and through mental health services for children and adolescents. Psychiatr Serv. 2003;54(1):60–6.

    Article  PubMed  Google Scholar 

  16. Pearson C, Janz T, Ali J. Mental and substance use disorders in Canada. Canada: Statistics Canada; 2013.

    Google Scholar 

  17. Urbanoski K, Inglis D, Veldhuizen S. Service use and unmet needs for substance use and mental disorders in Canada. Can J Psychiatr. 2017;62(8):551–9.

    Article  Google Scholar 

  18. MacDonald K, Fainman-Adelman N, Anderson KK, Iyer SN. Pathways to mental health services for young people: a systematic review. Soc Psychiatry Psychiatr Epidemiol. 2018;53(10):1005–38.

    Article  PubMed  PubMed Central  Google Scholar 

  19. Kowalewski K, McLennan JD, McGrath PJ. A preliminary investigation of wait times for child and adolescent mental health services in Canada. J Can Acad Child Adolesc Psychiatry. 2011;20(2):112–9.

    PubMed  PubMed Central  Google Scholar 

  20. Jordan G, Kinkaid M, Iyer SN, Joober R, Goldberg K, Malla A, et al. Baby or bathwater? Referrals of “non-cases” in a targeted early identification intervention for psychosis. Soc Psychiatry Psychiatr Epidemiol. 2018;53(7):757–61.

    Article  PubMed  Google Scholar 

  21. Malla A, Shah J, Iyer S, Boksa P, Joober R, Andersson N, et al. Youth mental health should be a top priority for health care in Canada. Can J Psychiatr. 2018;63(4):216–22.

    Article  Google Scholar 

  22. Iyer SN, Boksa P, Lal S, Shah J, Marandola G, Jordan G, et al. Transforming youth mental health: a Canadian perspective. Ir J Psychol Med. 2015;32(1):51–60.

    Article  Google Scholar 

  23. Repper J, Simpson A, Grimshaw G. Good practice guidance for involving carers and family members and close friends of service users in research. UK: NIHR Clinical Research Network: Mental Health; 2014. Available at:

  24. Canadian Institutes of Health Research. Canada’s Strategy for Patient-Orientated Research. Improving health outcomes through evidence-informed care. Ottawa: Canadian Institutes of Health Research; 2011. Available at:

    Google Scholar 

  25. Perestelo-Perez L, Gonzalez-Lorenzo M, Perez-Ramos J, Rivero-Santana A, Serrano-Aguilar P. Patient involvement and shared decision-making in mental health care. Curr Clin Pharmacol. 2011;6(2):83–90.

    Article  PubMed  Google Scholar 

  26. Tambuyzer E, Van Audenhove C. Is perceived patient involvement in mental health care associated with satisfaction and empowerment? Health Expect. 2015;18(4):516–26.

    Article  PubMed  Google Scholar 

  27. Malla A, Iyer S, McGorry P, Cannon M, Coughlan H, Singh S, et al. From early intervention in psychosis to youth mental health reform: a review of the evolution and transformation of mental health services for young people. Soc Psychiatry Psychiatr Epidemiol. 2016;51(3):319–26.

    Article  PubMed  Google Scholar 

  28. McGorry P, Bates T, Birchwood M. Designing youth mental health services for the 21st century: examples from Australia, Ireland and the UK. Br J Psychiatry Suppl. 2013;54:s30–5.

    Article  PubMed  Google Scholar 

  29. McGorry PD, Tanti C, Stokes R, Hickie IB, Carnell K, Littlefield LK, et al. Headspace: Australia’s National Youth Mental Health Foundation-where young minds come first. Med J Aust. 2007;187(7 Suppl):S68–70.

    PubMed  Google Scholar 

  30. Malla A, Iyer S, Shah J, Joober R, Boksa P, Lal S, et al. Canadian response to need for transformation of youth mental health services: ACCESS Open Minds (Esprits ouverts). Early Interv Psychiatry. 2019;13(3):697–706.

    Article  PubMed  PubMed Central  Google Scholar 

  31. Canadian Psychiatric Association. Wait time benchmarks for patients with serious psychiatric illnesses. Canada: Canadian Psychiatric Association; 2006.

    Google Scholar 

  32. Statistics Canada. Aboriginal Peoples in Canada: First Nations People, Metis and Inuit. National Household Survey, 2011. Canada: Statistics Canada;2013.

  33. Sawchuk J. Social conditions of indigenous peoples in Canada. The Canadian Encyclopedia. 2011; Available at:

  34. Baskin C. Structural determinants as the cause of homelessness for aboriginal youth. Crit Soc Work. 2007;8(1) Available at:

  35. Levin R, Herbert M. The experience of urban aboriginals with health care services in Canada: implications for social work practice. Soc Work Health Care. 2004;39(1–2):165–79.

    PubMed  Google Scholar 

  36. ACCESS Open Minds. Early identification: An evidence-based practical guide. (English and French). Montreal: ACCESS Open Minds; 2017. Available at:

  37. University of Kansas. Community Tool Box: Chapter 3, Section 8: Identifying Community Assets and Resources. Available at:

  38. Fawcett SB, Schultz JA, Holt CM, Collie-Akers V, Watson-Thompson J. Participatory research and capacity building for community health and development. J Prev Interv Community. 2013;41(3):139–41.

    Article  PubMed  Google Scholar 

  39. Holt CM, Fawcett SB, Schultz JA, Jones JA, Berkowitz B, Wolff TJ, et al. Disseminating online tools for building capacity among community practitioners. J Prev Interv Community. 2013;41(3):201–11.

    Article  PubMed  Google Scholar 

  40. ACCESS Open Minds. Community mapping guide: Conducting community mapping within the ACCESS Open Minds framework of youth mental health care. (English and French). Montreal: ACCESS Open Minds; 2017. Available at:

  41. Fortney JC, Unutzer J, Wrenn G, Pyne JM, Smith GR, Schoenbaum M, et al. A tipping point for measurement-based care. Psychiatr Serv. 2017;68(2):179–88.

    Article  PubMed  Google Scholar 

  42. Knaup C, Koesters M, Schoefer D, Becker T, Puschner B. Effect of feedback of treatment outcome in specialist mental healthcare: meta-analysis. Br J Psychiatry. 2009;195(1):15–22.

    Article  PubMed  Google Scholar 

  43. Zimmerman M, McGlinchey JB. Why don’t psychiatrists use scales to measure outcome when treating depressed patients? J Clin Psychiatry. 2008;69(12):1916–9.

    Article  PubMed  Google Scholar 

  44. Statistics Canada. Canadian community health survey (CCHS) – mental health user guide. Ottawa: Statistics Canada; 2013.

  45. Statistics Canada. Aboriginal peoples survey (APS). Ottawa: Statistics Canada; 2012.

  46. Hussey MA, Hughes JP. Design and analysis of stepped wedge cluster randomized trials. Contemp Clin Trials. 2007;28(2):182–91.

    Article  PubMed  Google Scholar 

  47. Hemming K, Haines TP, Chilton PJ, Girling AJ, Lilford RJ. The stepped wedge cluster randomised trial: rationale, design, analysis, and reporting. BMJ. 2015;350:h391.

    Article  CAS  PubMed  Google Scholar 

  48. Andersson N, Shea B, Amaratunga C, McGuire P, Sioui G. Rebuilding from resilience: research framework for a randomized control trial of community-led interventions to prevent domestic violence in aboriginal communities. Pimatisiwin. 2010;8(2):61–88.

    PubMed  PubMed Central  Google Scholar 

  49. Brown CA, Lilford RJ. The stepped wedge trial design: a systematic review. BMC Med Res Methodol. 2006;6:54.

    Article  PubMed  PubMed Central  Google Scholar 

  50. Haines TP, Hemming K. Stepped-wedge cluster-randomised trials: level of evidence, feasibility and reporting. J Physiother. 2018;64(1):63–6.

    Article  PubMed  Google Scholar 

  51. Grayling MJ, Wason JM, Mander AP. Stepped wedge cluster randomized controlled trial designs: a review of reporting quality and design features. Trials. 2017;18(1):33.

    Article  PubMed  PubMed Central  Google Scholar 

  52. Barker D, McElduff P, D’Este C, Campbell MJ. Stepped wedge cluster randomised trials: a review of the statistical methodology used and available. BMC Med Res Methodol. 2016;16:69.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  53. Chan AW, Tetzlaff JM, Altman DG, Laupacis A, Gotzsche PC, Krleza-Jeric K, et al. SPIRIT 2013 statement: defining standard protocol items for clinical trials. Ann Intern Med. 2013;158(3):200–7.

    Article  PubMed  PubMed Central  Google Scholar 

  54. Chan AW, Tetzlaff JM, Gotzsche PC, Altman DG, Mann H, Berlin JA, et al. SPIRIT 2013 explanation and elaboration: guidance for protocols of clinical trials. BMJ. 2013;346:e7586.

    Article  PubMed  PubMed Central  Google Scholar 

  55. ACCESS Open Minds. ACCESS Open Minds- A transformation in youth mental health;2017. Available at:

  56. ACCESS Open Minds. ACCESS Open Minds- Informed consent (English);2017. Available at:

  57. ACCESS Open Minds. ACCESS Esprits ouverts- Consentement informé (French);2017. Available at:

  58. Bhui K, Mohamud S, Warfa N, Craig TJ, Stansfeld SA. Cultural adaptation of mental health measures: improving the quality of clinical practice and research. Br J Psychiatry. 2003;183:184–6.

    Article  PubMed  Google Scholar 

  59. World Health Organization. Management of substance abuse. Process of translation and adaptation of instruments;2019. Available at:

  60. Statistics Canada. Visible minority and population group reference guide, National Household Survey, 2011. Ottawa: Ministry of Industry; 2013.

  61. Dennis ML, Chan YF, Funk RR. Development and validation of the GAIN short screener (GSS) for internalizing, externalizing and substance use disorders and crime/violence problems among adolescents and adults. Am J Addict. 2006;15(Suppl 1):80–91.

    Article  PubMed  PubMed Central  Google Scholar 

  62. Posner K, Brown GK, Stanley B, Brent DA, Yershova KV, Oquendo MA, et al. The Columbia-suicide severity rating scale: initial validity and internal consistency findings from three multisite studies with adolescents and adults. Am J Psychiatry. 2011;168(12):1266–77.

    Article  PubMed  PubMed Central  Google Scholar 

  63. Kessler RC, Andrews G, Colpe LJ, Hiripi E, Mroczek DK, Normand SL, et al. Short screening scales to monitor population prevalences and trends in non-specific psychological distress. Psychol Med. 2002;32(6):959–76.

    Article  CAS  PubMed  Google Scholar 

  64. Fleishman JA, Zuvekas SH. Global self-rated mental health: associations with other mental health measures and with role functioning. Med Care. 2007;45(7):602–9.

    Article  PubMed  Google Scholar 

  65. Zajacova A, Dowd JB. Reliability of self-rated health in US adults. Am J Epidemiol. 2011;174(8):977–83.

    Article  PubMed  PubMed Central  Google Scholar 

  66. Mawani FN, Gilmour H. Validation of self-rated mental health. Health Rep. 2010;21(3):61–75.

    PubMed  Google Scholar 

  67. Rickwood DJ, Mazzer KR, Telford NR, Parker AG, Tanti CJ, McGorry PD. Changes in psychological distress and psychosocial functioning in young people visiting headspace centres for mental health problems. Med J Aust. 2015;202(10):537–42.

    Article  PubMed  Google Scholar 

  68. Hilferty F, Cassells R, Muir K, Duncan A, Christensen D, Mitrou F, et al. Is headspace making a difference to young people’s lives? Final report of the independent evaluation of the headspace program. Sydney: Social Policy Research Centre, UNSW Australia; 2015.

    Google Scholar 

  69. Berk M, Ng F, Dodd S, Callaly T, Campbell S, Bernardo M, et al. The validity of the CGI severity and improvement scales as measures of clinical effectiveness suitable for routine clinical use. J Eval Clin Pract. 2008;14(6):979–83.

    Article  PubMed  Google Scholar 

  70. Bunlop BW, Gray J, Rapaport MH. Transdiagnostic clinical global impression scoring for routine clinical settings. Behav Sci (Basel). 2017;7(3):40.

    Google Scholar 

  71. Hilsenroth MJ, Ackerman SJ, Blagys MD, Baumann BD, Baity MR, Smith SR, et al. Reliability and validity of DSM-IV axis V. Am J Psychiatry. 2000;157(11):1858–63.

    Article  CAS  PubMed  Google Scholar 

  72. Miller SD, Duncan SL, Brown J, Sparks JA, Claud DA. The outcome rating scale: a preliminary study of reliability, validity, and feasibility of a brief visual analog measure. Journal of Brief Therapy. 2003;2:91–100.

    Google Scholar 

  73. Skevington SM, Lofty M, O’Connell KA. WHOQOL group. The World Health Organization’s WHOQOL-BREF quality of life assessment: psychometric properties and results of the international field trial. A report from the WHOQOL group. Qual Life Res. 2004;13(2):299–310.

    Article  CAS  PubMed  Google Scholar 

  74. Law D, Jacob J. Goals and goal based outcomes (GBOs) some useful information. 3rd ed. London: CAMHS Press; 2015.

    Google Scholar 

  75. Abrines-Jaume N, Midgley N, Hopkins K, Hoffman J, Martin K, Law D, et al. A qualitative analysis of implementing shared decision making in child and adolescent mental health services in the United Kingdom: stages and facilitators. Clin Child Psychol Psychiatry. 2016;21(1):19–31.

    Article  PubMed  Google Scholar 

  76. Wolpert M, Ford T, Trustam E, Law D, Deighton J, Flannery H, et al. Patient-reported outcomes in child and adolescent mental health services (CAMHS): use of idiographic and standardized measures. J Ment Health. 2012;21(2):165–73.

    Article  PubMed  Google Scholar 

  77. Jacob J, De Francesco D, Deighton J, Law D, Wolpert M, Edbrooke-Childs J. Goal formulation and tracking in child mental heath settings: when is it more likely and is it associated with satisfaction with care? Eur Child Adolesc Psychiatry. 2017;26(7):759–70.

    Article  PubMed  PubMed Central  Google Scholar 

  78. Rush B, Hannson E, Cvetanova Y, Rotondi N, Furlong A, Behrooz R. Development of a client perception of care tool for mental health and addictions: Qualitative, quantitative, and psychometric analysis. Final Report for Ministry of Health and Long-Term Care. Toronto: Health Systems and Health Equity Research, Centre for Addiction and Mental Health; 2013.

    Google Scholar 

  79. Walker JS, Powers LE. Introduction to the youth self-efficacy scale/mental health and the youth participation in planning scale. Portland: Research and Training Centre on Family Support and Children’s Mental Health, Portland State University; 2007.

    Google Scholar 

  80. Campbell A, Hemsley S. Outcome rating scale and session rating scale in psychological practice: clinical utility of ultra-brief measures. Clin Psychol. 2009;13:1–9.

    Article  Google Scholar 

  81. Reid RJ. Embedding research in the learning health system. HealthcarePapers 16:30–35;doi:

  82. U.S. Food and Drug Administration. Regulations: Good Clinical Practice and Clinical Trials. Washington: U.S. Food and Drug Administration. Available at:

  83. Royal Commission on Aboriginal Peoples. Report of the Royal Commission on Aboriginal Peoples. Ottawa: Canada Communication Group—Publishing; 1996. Available at:

    Google Scholar 

  84. The First Nations Information Governance Centre. Ownership, Control, Access and Possession (OCAP™): The Path to First Nations Information Governance. May 2014. (Ottawa: The First Nations Information Governance Centre, May 2014)..Available at

  85. Canadian Institutes of Health Research, Natural Sciences and Engineering Research Council of Canada, and Social Sciences and Humanities Research Council of Canada, Tri-Council Policy Statement: Ethical Conduct for Research Involving Humans; December 2014. Chapter 9: Research Involving the First Nations, Inuit and Metis People of Canada. Available at:

  86. Government of Canada. Jordan’s Principle. Ottawa: Government of Canada. Available at:

  87. Statistics Canada. Table 105-1101-mental health profile, Canadian community health survey- mental health (CCHS), by age group and sex, Canada and provinces, occasional (number unless otherwise noted), CANSIM (database). Ottawa: Statistics Canada Available at:

  88. Statistics Canada. Census profile, 2016 census. Ottawa: Statistics Canada; 2017. Available at:

    Google Scholar 

  89. Iyer SN, Boksa P, Joober R. How youth mental healthcare is being transformed in diverse settings across Canada: Reflections on the experience of the ACCESS Open Minds network. Early Interv Psychiatry. 2019;13(S1):8–11.

    Article  PubMed  Google Scholar 

  90. Abdel-Baki A, Aubin D, Morisseau-Guillot R, Dupont ME, Bauco P, Lal S, Shah J, Joober R, Boksa P, Malla A, Iyer SN. Improving Mental Health Services for Homeless Youth in Downtown Montreal, Canada: Partnership between a Local Network and ACCESS Open Minds, a National Service Transformation Research Initiative. Early Interv Psychiatry. 2019;13(S1):20–28.

    Article  PubMed  PubMed Central  Google Scholar 

  91. Hutt-MacLeod D, Rudderham H, Sylliboy A, Sylliboy-Denny M, Liebenberg L, Denny JF, Gould MR, Gould N, Nossal M, Iyer S, Malla A, Boksa P. Eskasoni First Nation’s transformation of youth mental healthcare: Partnership between a Mi’kmaq community and the ACCESS Open Minds research project in implementing innovative practice and service evaluation. Early Interv Psychiatry. 2019;13(S1):42–47.

    Article  PubMed  PubMed Central  Google Scholar 

  92. Reaume-Zimmer P, Chandrasena R, Malla A, Joober R, Boksa P, Shah JL, Iyer SN, Lal S. Transforming Youth Mental Health Services in a Semi-Urban and Rural Region of Canada: ACCESS Open Minds Chatham-Kent. Early Interv Psychiatry. 2019;13(S1):48–55.

    Article  PubMed  PubMed Central  Google Scholar 

  93. Etter M, Goose A, Nossal M, Chishom-Nelson J, Heck C, Joober R, Boksa P, Lal S, Shah J, Andersson N, Iyer S, Malla A (2019). Improving youth mental wellness services in an Indigenous context in Ulukhaktok, Northwest Territories: ACCESS Open Minds project. Early Interv Psychiatry. 2019;13(S1):35–41.

    Article  PubMed  PubMed Central  Google Scholar 

  94. Abba-Aji A, Hay K, Kelland J, Mummery C, Urichuk L, Snaterse M, Chue P, Lal S, Boksa P, Joober R, Malla A, Iyer S, Shah JL. Youth mental health service transformation in a large urban centre: ACCESS Open Minds Edmonton. Early Interv Psychiatry. 2019;13(S1):14–19.

    Article  PubMed  PubMed Central  Google Scholar 

  95. Vallianatos H, Friese K, Perez JM, Slessor J, Thind R, Dunn J, Lal S, Boksa P, Joober R, Malla A, Iyer SN, Shah JL. ACCESS Open Minds University of Alberta: Transforming student mental health services across a large post-secondary educational institution. Early Interv Psychiatry. 2019;13(S1):56–64.

    Article  PubMed  PubMed Central  Google Scholar 

  96. Dubé A, Lancu P, Doucet D, Tranchant CC, Joachin A, Malchow J, Robichaud S, Hach M, Godin I, Bourdon L, Bourque J, Iyer S, Malla A, Beaton AM. Transforming child and youth mental healthcare: ACCESS Open Minds New Brunswick in the rural Francophone region of the Acadian Peninsula. Early Interv Psychiatry. 2019;13(S1):29–34.

    Article  PubMed  PubMed Central  Google Scholar 

  97. El Emam K, Jonker E, Sampson M, Krleza-Jeric K, Neisa A. The use of electronic data capture tools in clinical trials: web-survey of 259 Canadian trials. J Med Internet Res. 2009;11(1):e8.

    Article  PubMed  PubMed Central  Google Scholar 

Download references


We acknowledge the valuable contributions of the many additional ACCESS Open Minds Network members, not listed as authors on this manuscript, who have collaborated on this project and whose extensive efforts have contributed to its quality and spirit. The network includes a vibrant National Youth Council, a Family and Carers Council and an Indigenous Council. The corresponding author can be contacted for a list of all network members.

Availability of data and material

Not applicable for current article. Data from the study will only be accessible to study investigators. Its use and access will follow OCAP principles and signed data sharing agreements between the ACCESS Open Minds central office at the Douglas Mental Health University Institute and individual participating sites. Please contact corresponding author for any request related to accessing data from this project.


ACCESS Open Minds is funded as a Strategy for Patient-Oriented Research (SPOR) of the Canadian Institutes of Health Research (CIHR), with joint funding from CIHR and the Graham Boeckh Foundation. The funders play no direct role in the design of the study or the collection, analysis, and interpretation of the data.

Author information

Authors and Affiliations



SNI, JS, PB, SL, RJ, NA, RF, AAB, AB, PRZ, DHM, RC, MAL, CR, JT, ME, HV, AAA, SB, AM & AKM took part in study conception and design. SNI, JS, PB, SL, RJ, NA, RF, AAB, AB, PRZ, DHM, RC, MAL, CR, JT, ME, HV, AAA, SB, AM & AKM were involved in drafting the work or revising it critically for important intellectual content. All authors approved the submitted version. SNI and AKM were senior leaders as the Scientific-Clinical Director and the Nominated Principal Investigator of the ACCESS Open Minds project.

Corresponding author

Correspondence to Srividya N. Iyer.

Ethics declarations

Ethics approval and consent to participate

Research Ethics Board approval was received from the Douglas Mental Health University Institute (IUSMD-15-21), followed by institutional ethics approval from participating ACCESS Open Minds sites, where needed. Any important modifications to protocol will be notified to the Douglas ethics board, ethics boards for other institutions pertaining to participating sites, trial registry, funding bodies and the journal in which the protocol is published.

At all sites, youth above the age of 18 are approached for written informed consent. For minors below the age of 14 or youth with diminished capacity, consent is sought from a parent/legal representative along with assent from the young person. Provincial/institutional regulations vary in terms of policy around the need for consent from a parent/legal representative in the case of youth between the ages of 14 and 18 (see Additional file 4).

Consent for publication

Not applicable.

Competing interests

Ridha Joober is a speaker and/or consulting committee member for Janssen, Lundbeck, Myelin, Otsuka, Perdue, Pfizer, Shire, and Sunovian; has received grants from Astra Zeneca, BMS, HLS, Janssen, Lundbeck, and Otsuka; and has royalties from Henry Stewart talks. Ashok K. Malla reports research funding for an investigator-initiated project from BMS Canada and honoraria for lectures and consulting activities with Lundbeck and Otsuka. For both authors, none of the funds received have any relation to this manuscript. The other authors report no competing interests.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Additional files

Additional file 1:

SPIRIT 2013 checklist for ACCESS Open Minds. (DOCX 53 kb)

Additional file 2:

Model of consent form used in the ACCESS Open Minds project. (DOCX 54 kb)

Additional file 3:

Psychometric properties of study instruments. (DOCX 20 kb)

Additional file 4:

Listing of ethics boards; age of research consent and need for parental consent for each site in the ACCESS Open Minds project. (DOCX 15 kb)

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (, which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver ( applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Iyer, S.N., Shah, J., Boksa, P. et al. A minimum evaluation protocol and stepped-wedge cluster randomized trial of ACCESS Open Minds, a large Canadian youth mental health services transformation project. BMC Psychiatry 19, 273 (2019).

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: