Skip to main content
  • Research article
  • Open access
  • Published:

Implementation of internet-delivered cognitive behavior therapy within community mental health clinics: a process evaluation using the consolidated framework for implementation research



Depression and anxiety are prevalent and under treated conditions that create enormous burden for the patient and the health system. Internet-delivered cognitive behavior therapy (ICBT) improves patient access to treatment by providing therapeutic information via the Internet, presented in sequential lessons, accompanied by brief weekly therapist support. While there is growing research supporting ICBT, use of ICBT within community mental health clinics is limited. In a recent trial, an external unit specializing in ICBT facilitated use of ICBT in community mental health clinics in one Canadian province (ISRCTN42729166; registered November 5, 2013). Patient outcomes were very promising and uptake was encouraging. This paper reports on a parallel process evaluation designed to understand facilitators and barriers impacting the uptake and implementation of ICBT.


Therapists (n = 22) and managers (n = 11) from seven community mental health clinics dispersed across one Canadian province who were involved in implementing ICBT over ~2 years completed an online survey (including open and closed-ended questions) about ICBT experiences. The questions were based on the Consolidated Framework for Implementation Research (CFIR), which outlines diverse constructs that have the potential to impact program implementation.


Analyses suggested ICBT implementation was perceived to be most prominently facilitated by intervention characteristics (namely the relative advantages of ICBT compared to face-to-face therapy, the quality of the ICBT program that was delivered, and evidence supporting ICBT) and implementation processes (namely the use of an external facilitation unit that aided with engaging patients, therapists, and managers and ICBT implementation). The inner setting was identified as the most significant barrier to implementation as a result of limited resources for ICBT combined with greater priority given to face-to-face care.


The results contribute to understanding facilitators and barriers to using ICBT within community mental health clinics and serve to identify recommendations for improving uptake and implementation of ICBT in clinic settings.

Peer Review reports


Despite being highly prevalent and disabling conditions [1], depression and anxiety often remain untreated [2]. Barriers to treatment are diverse and include limited perceived need for treatment, preference to independently manage problems, inadequate finances, time constraints, transportation or mobility challenges, poor access to providers, or concerns about privacy and stigma [2]. Internet-delivered cognitive behavior therapy (ICBT) for depression and anxiety has the potential to address many of these barriers and improve patient access to evidence-based treatment [3]. ICBT involves patients reviewing structured weekly lessons delivered via the Internet over the course of several months. Homework is assigned to facilitate learning and brief therapist support is typically provided using emails or telephone calls. Randomized controlled trials provide strong support for ICBT with large treatment effects identified for those receiving ICBT [4]; these promising findings extend to when ICBT is delivered in routine practice settings [5]. Nevertheless, this latter research primarily has been conducted in specialized ICBT clinics, where there are limited competing demands on therapists’ time [5].

At present, ICBT is largely unavailable on a routine basis to most Canadian patients who have depression and anxiety [3]. In 2014, the Mental Health Commission of Canada released a briefing document encouraging broader use of technology in mental health care [6]. Consistent with this call, a recent trial by our group compared ICBT for depression and anxiety when delivered by therapists working in a specialized clinic to when ICBT was delivered by therapists working in community mental health clinics distributed across one Canadian province (i.e., clinics where therapists primarily deliver face-to-face care but secondarily use ICBT) [7]. Supporting the public health potential of ICBT, regardless of setting, completion rates, satisfaction, and outcomes were very strong. In both the specialized clinic and community mental health clinics, 78% of patients completed all core ICBT lessons, over 94% reported that the course was worth their time, and large effects were found from pre- to post-treatment on symptoms of depression and generalized anxiety. Uptake of ICBT by the clinic therapists was encouraging, with continued use of ICBT within the clinics post-trial and growth in use of ICBT over time (i.e., 110 patients were treated by community therapists in first 12 months the ICBT program was implemented; 205 patients were treated by community therapists over the last 12 months).

While there is growing literature on attitudes towards ICBT [8, 9] and the effectiveness of ICBT in clinical practice settings [5], as well as on factors that impact the implementation of e-health generally [10], there is limited literature concerning models of disseminating ICBT and experiences with implementing ICBT. Lack of information on implementation could represent a barrier to wider scale implementation. In the present research, we first describe the facilitation model and strategies that were used [11] to support ICBT implementation in community mental health clinics in Saskatchewan. Second, we use the Consolidated Framework for Implementation Research CFIR; [12] to examine the perceived barriers and facilitators to ICBT implementation among therapists and managers within the community mental health clinics. The CFIR is a comprehensive meta-theoretical framework (based on a review of 19 other theories) that outlines diverse constructs that have the potential to impact program implementation. More specifically, this study explores the extent to which ICBT implementation was impacted by intervention characteristics (e.g., evidence, advantages, complexity, quality, cost), the outer setting (e.g., patient needs, networks, policies), the inner setting (e.g., structure, communication, culture), individual characteristics (e.g., therapist/manager knowledge, self-efficacy, interest), and implementation processes (e.g., planning, engaging, conducting, reflecting, and evaluating). The findings may assist others who are interested in incorporating ICBT into service delivery within community mental health clinics in that we highlight a process for ICBT implementation. We also identify facilitators that are associated with the uptake of ICBT as well as barriers that need to be overcome to improve implementation efforts. The results also contribute to growing literature on CFIR and how it can be used to understand and evaluate program implementation efforts [13].



This process evaluation aimed to identify barriers and facilitators that influenced ICBT implementation in seven community mental health clinics distributed across one province. A bespoke survey, designed around CFIR constructs, consisting of open- and closed-ended questions, was administered to therapists and managers approximately 2 years after ICBT for depression and anxiety was implemented in the clinics.

Context and setting

Geographical context

This study took place in Saskatchewan, Canada, which is a landlocked province located in west-Central Canada, and has an area of ~650,000 km2. The population estimate in 2016 was ~1.1 million with approximately half of the population residing in two large cities over 250,000, and other residents living in small cities, towns, or rural and remote areas [14]. Most residents live in the southern prairie half of the province, while the northern forested part of the province is sparsely populated. The majority of Saskatchewan residents identify as Caucasian.

Healthcare system

Throughout Canada, medically necessary medical office and hospital care is publicly funded and financed by the government through taxation [15]. In addition, mental health care that is delivered within publicly funded healthcare institutions (e.g., hospitals, community mental health clinics) is also covered. Nevertheless, resources allocated to mental health care is lower than the demand for these services with many patients reporting their needs are unmet or only partially met [16]. Many individuals in Canada turn to private health insurance or personally pay for mental health services [15].

The community mental health clinics involved in the current research were funded by the provincial government to provide outpatient mental health services to patients within specific regions of the province. Table 1 describes the population served by the clinics (ranging from 31, 663 to 26, 3065 residents), the length of time clinics had delivered the ICBT at the time of the survey (19–29 months), the number of patients from each health region who received ICBT (36–386), the number of therapists from each clinic who delivered ICBT (1–13), and the number of therapists and managers from each clinic who participated in the study. Table 2 describes the external facilitation model that was used to actively aid ICBT implementation within the community clinics. Specifically, we elaborate on how a specialized ICBT unit (Online Therapy Unit; OTU) at the University of Regina facilitated ICBT implementation within the clinics, such as by developing a website and policies and procedures for delivering ICBT, training therapists in ICBT, and auditing and providing clinical feedback to therapists. Of note, OTU therapists also treated patients with ICBT, taking on patients when community clinic therapists were unavailable. At the time of the study, 385 patients had been treated by community therapists and 479 had been treated by OTU therapists.

Table 1 Population served, months offering ICBT, ICBT utilization and participation rate by community mental health clinic
Table 2 External facilitation model used to facilitate ICBT implementation within community mental health clinics

Online therapy process

Table 3 describes how patients were deemed eligible for ICBT, how patients were assigned to community therapists or OTU therapists, and the specifics of the ICBT program utilized. In brief, after screening, patients were assigned to the first available therapist. The ICBT program was then accessed by patients and therapists by logging onto a secure server. On this server, ICBT materials were viewed, psychological measures were administered to track patient progress through treatment, and patients and therapists communicated via secure email. The ICBT program materials were developed by the eCentreClinic ( at Macquarie University, Sydney, Australia and are available through a license to the Unit. The ICBT course contains five core lessons, which patients review over 8 weeks. Using text and images presented in a slideshow format, the lessons describe: 1) the cognitive behavioral model and symptoms of depression and anxiety; 2) thought monitoring and challenging; 3) de-arousal strategies and pleasant activity scheduling; 4) graduated exposure; and, 5) relapse prevention. At the end of the lesson, patients are able to download a summary of the materials. Therapists also encourage patient completion of homework assignments to facilitate skill acquisition. Furthermore, patients read case studies that serve to demonstrate the application of skills. On a weekly basis, therapists are required to check in on patients and assist them in learning and applying the skills taught in the course. Most contact is via secure email, but phone calls can also be made if this is preferred by patients or clinically indicated.

Table 3 Overview of ICBT Delivery Process


Participants included managers and therapists who worked in seven community mental health clinics across Saskatchewan who were involved in ICBT implementation at the time this survey was conducted. Completion of the survey was high, with 11 out of 12 managers (n = 3 psychologists; n = 5 social workers; n = 2 nurses; n = 1 administrators; n = 2 male; n = 9 female) and 22 out of 23 therapists (n = 7 psychologists; n = 11 social workers; n = 3 nurses; n = 1 counselor; n = 2 male; n = 20 female) who were contacted agreeing to participate. On average, therapists who participated had provided ICBT to 11.41 (SD = 7.12) patients at the time they completed the survey. Of note, we did not contact 17 therapists who provided ICBT in the past, but had retired or resigned from their position at the time of the survey.


Research ethics board approval for the study was obtained from the University of Regina. All participants provided electronic informed consent and were offered a $10 coffee card as a token of appreciation. The secure online survey (available in Additional file 1) was administered using the Qualtrics software program. After completing background questions, participants rated their perceptions of the importance of ICBT (e.g., whether residents should have access to ICBT, whether involvement with ICBT implementation was worth their time) on a scale from 1 “strongly disagree” to 5 “strongly agree”. Participants were then asked their opinions regarding the number of patients who should receive ICBT (i.e., “many more”, “somewhat more”, “the same number”, “somewhat fewer”, or “none”). Participants were then provided with a description of each CFIR domain (intervention characteristics, outer setting, inner setting, individual characteristics, and implementation process) and were asked to comment on both positive and negative aspects of the domain on ICBT implementation. For example, the following question was posed: “The inner clinic setting (e.g., structure, communication, culture) can influence program implementation efforts. Please comment on any positive or negative factors within your clinic setting that you feel may have influenced the implementation of ICBT within your health region.” After providing open-ended feedback, participants rated agreement with a series of 47 statements (see Table 5 for items) related to ICBT implementation on a scale from 1 (“strongly disagree”) to 5 (“strongly agree”). The statements referenced facets of each CFIR domain. Example questions included: “It is positive that the ICBT Wellbeing Course was developed externally and the health region did not have to develop our own ICBT program” (Intervention Characteristics); “My health care region is aware of the high need for mental health care” (Outer Setting); “My health region has an adequate number of therapists available to deliver ICBT” (Inner Setting); “Therapists in my health region have adequate knowledge of ICBT” (Individual Characteristics); and, “We spent adequate time planning how to deliver ICBT in advance in my health region” (Implementation Processes).


Responses to open-ended questions were exported from Qualtrics into NVivo Version 10, a qualitative data software tool. The CFIR constructs served as a coding framework with text classified into pre-existing categories (see for definitions) using deductive principles [17] to identify the presence of such categories in the dataset. Two coders read and coded data together reaching consensus on the CFIR domain and construct. Each sentence could contain multiple codes. Alternatively, concurrent sentences of the same thematic code could be conjoined into one unit. Once initial coding was complete, the coders examined quotes assigned to each construct as a whole and any quotes that were identified as inconsistent with the designated construct were re-examined to determine the most appropriate construct. A third researcher then reviewed quotes representing each construct and identified any points of disagreement that were then resolved through consensus. In addition to coding CFIR constructs, the coders classified comments as either positive (facilitator) or negative (barrier) in tone. Following all coding, NVivo’s query function was used to calculate the total frequency of each construct. The frequency score represents the total number of times the construct was present in the survey (see Table 4 for frequencies and representative quotes). At times, participants made recommendations to improve ICBT implementation. These comments were also classified according to domain to identify CFIR domains requiring improvement. Descriptive statistics were calculated to examine responses to rating scale survey questions (see Table 5). Ratings above 4 were interpreted as positive perceptions of implementation, while ratings below 3 were interpreted as areas of potential improvement.

Table 4 Frequency of positive and negative comments about implementation of internet-delivered cognitive behaviour therapy
Table 5 Therapist and manager agreement that ICBT implementation was facilitated by CFIR constructs


Perceptions of ICBT

There was strong indication that both therapists and managers had positive perceptions of ICBT. Specifically, both agreed strongly that Saskatchewan residents should have access to ICBT (M = 4.61/5, SD = .50), that health regions should be committed to ensuring access to ICBT (M = 4.12/5; SD = 1.05), that health regions should identify barriers and facilitators associated with ICBT (M = 4.12/5; SD = 1.05), that health regions should continuously monitor and evaluate ICBT (M = 4.06; SD = 1.25) and, that it was worth their time to be involved in ICBT (M = 4.39; SD = .79). Also reflecting positive perceptions of ICBT, ~45% of participants indicated that a somewhat greater number of patients should receive ICBT in the future, ~21% supported treating about the same number of patients, and promisingly ~15% reported that many more patients should receive ICBT. Very few participants indicated that the health regions were treating too many patients (9.1%) or should not provide ICBT (9.1%).

Perceptions of implementation

Table 4 lists the frequency of positive and negative comments along with representative quotes made by participants related to each CFIR construct. Table 5 displays mean ratings on statements related to the CFIR constructs. The findings are summarized below by CFIR domain with attention given to recommendations for improving implementation as well as differences between therapists and managers.

Intervention characteristics

Qualitative findings

Overall, 26% (75/292) of participant responses concerned the impact of the intervention on implementation; the majority of these comments were positive (65%; 49/75). Both therapists and managers emphasized the relative advantages of ICBT over face-to-face care, noting it overcomes multiple barriers to face-to-face therapy related to time and location, and is a more convenient method of receiving care. Many participants also highlighted that the design and packaging of the ICBT program was excellent (e.g., brief, organized, clear, practical, user-friendly) and recognized the program as evidence-based.

The most common barrier related to the intervention concerned the lack of adaptability of the ICBT program for certain patients. Therapists reported that the ICBT program is highly standardized and not designed for blending with face-to-face care or adapted for patients who need more or less support or time to complete the program. Many participants shared it would be helpful to offer ICBT for other conditions, such as addiction and trauma. As stated by one participant: “This [ICBT] needs to continue, we need to explore other areas where we could expand this type of service broader” (Manager).

Quantitative findings

The mean rating for questions assessing intervention characteristics was above 4.5, also suggesting that participants viewed intervention characteristics as facilitating implementation. As with qualitative feedback, participants rated the ICBT program as having strong research evidence, multiple advantages, and being well designed and packaged. Ratings also revealed that participants felt it was beneficial that the ICBT program was externally developed, could be used to treat both depression and anxiety in a short period of time, did not result in additional costs to the health region, and could be implemented on a small scale.

Outer setting

Qualitative findings

Relative to other CFIR constructs, participants made fewer comments about how the outer setting impacted ICBT implementation (~5%; 15/292); 73% (11/15) of the comments made were positive, with many participants reporting that ICBT aligned with patient needs for resources, and thus facilitated implementation. Participants highlighted that is helpful to have another treatment option for patients, especially when patients are waiting for face-to-face services, or when patients are reluctant to seek in-person services related to stigma or inconvenience. There was recognition among some participants that ICBT would not meet the needs of all patients who have a strong preference for face-to-face care.

Quantitative findings

Ratings were consistent with qualitative comments and suggested that the outer setting had somewhat aided implementation. Ratings indicated that participants were aware of the high need for mental health care, and health regions benefit from partnering with the OTU. Other health regions were not regarded as exerting pressure to implement ICBT.

Inner setting

Qualitative findings

When queried about experiences implementing ICBT, participants most frequently described the impact of the inner setting on implementation (35% of all comments; 101/292). The majority of these comments (~74%; 75/101) were negative in tone suggesting the inner setting likely hindered ICBT implementation. Participants described low readiness for implementation due to lack of available resources. Some therapists discussed feeling pressured to deliver ICBT and that ICBT added to their current high caseloads resulting in an unmanageable workload. Several indicated that they would often deliver ICBT during lunch hours or after work. Managers echoed therapists’ sentiments, citing staff shortages and therapists’ high existing workloads as barriers to implementing ICBT. Implementation climate also was raised as a barrier to implementation with therapists often discussing the lower relative priority of ICBT compared to face-to-face therapy. As described by one manager, there is a culture that perceives face-to-face counseling as the optimal service.

While the inner setting was described primarily as a barrier to ICBT implementation, some participants reported that leadership engagement with ICBT facilitated implementation (e.g., “A positive factor is that there is support from management to implement ICBT” Therapist). It was also noted that having access to knowledge about ICBT through the OTU positively impacted implementation, such as attending a one-day workshop on ICBT prior to delivering ICBT and also being able to consult with the Unit on an ongoing basis to address any challenges in delivering ICBT. Additionally, some participants reported implementation was facilitated by the compatibility of ICBT with the clinic need to manage waiting lists and provide stepped care.

Some participants offered recommendations for improving the inner setting to facilitate ICBT. Numerous participants indicated that resources for ICBT need to be addressed, such as ensuring adequate time for ICBT (e.g., reducing face-to-face therapy patients), creating positions that predominantly deliver ICBT or, conversely, training more therapists in ICBT to reduce the ICBT workload. Several participants indicated increasing the ICBT patient load (while reducing face-to-face care) would improve therapist self-efficacy and thus ICBT effectiveness and efficiency. As another method of improving the inner setting, some participants recommended therapists be rewarded for providing ICBT (e.g., educational opportunities, monetary reward).

Quantitative findings

On rating scales, the average rating on inner setting items was 3.32, indicating primarily neutral impressions of the impact of the inner setting on ICBT implementation. Participants gave lower ratings to items regarding: having adequate time to reflect on ICBT and address challenges, setting specific goals and priority for ICBT, therapists being recognized for their work, ease of incorporating ICBT into the clinic workflow, adequacy of number of therapists to deliver ICBT, and adequacy of number of meetings and time to learn and discuss ICBT. Reflecting a positive tone, similar to qualitative comments, participants agreed that having adequate access to information on ICBT benefitted implementation.

Individual characteristics

Qualitative findings

Participants made relatively few comments about the impact of individual characteristics on implementation (13% of comments; 37/292); the comments were both negative (56%; 21/37) and positive (44%; 16/37). Comments reflected the perception that therapists, who had positive beliefs about ICBT, facilitated implementation, while therapists who held unenthusiastic beliefs about ICBT (e.g., viewing ICBT delivery as “boring” and “unrewarding”) hindered program uptake. Managers shared that staff who reported uncertainty regarding ICBT could have inadvertently transferred their uncertainty to their patients.

Quantitative findings

In terms of ratings, participants generally agreed that therapists had adequate knowledge about ICBT, were competent in ICBT, were committed to improving their clinics, have strong computer skills, and have a strong interest in learning. Less certainty was expressed regarding therapists’ commitment to offering ICBT.

Implementation processes

Qualitative findings

Approximately, 22% (64/292) of comments made by participants concerned implementation processes; the majority of these comments (64%; 41/64) were positive, especially in terms of planning, executing, and reflecting and evaluating. However, engagement was perceived as both facilitating and hindering implementation. For example, participants reported that ICBT was facilitated by training therapists to deliver ICBT and by educating patients and referring-providers about ICBT. Furthermore, a number of participants mentioned that “champions” who were keen to deliver ICBT facilitated implementation. Other participants noted that the promotion of ICBT to patients took considerable time. Managers’ responses reflected having difficulty engaging some therapists, referring-providers (e.g., family physicians, psychiatrists), and the general public with ICBT.

In terms of recommendations, the majority of participants indicated that ICBT requires additional promotion to the public (i.e., potential patients) and to referring-providers. Several participants also noted the need to correct misperceptions that face-to-face services are always superior to ICBT. One participant expressed, “We need to continue to have conversations with staff about its effectiveness, ensuring staff have a good understanding of the program” (Manager). In terms of ICBT training, participants recommended creating an online training program for therapists to increase the ease of access to training, and ultimately the uptake of ICBT.

Other recommendations participants shared included adjusting the model of service delivery. Some voiced that ICBT should be offered routinely to patients who are waiting or as a first step in care before face-to-face care. Others suggested alternative models of delivery to facilitate implementation, such as ICBT being provided by one stand-alone organization: “Instead of spreading this service across all of the different health regions, I think it would be more efficient to fund/resource one centralized service line to provide ICBT province-wide” (Manager). Lastly, participants expressed an interest in receiving more comprehensive updates on patient utilization of ICBT in an aesthetically pleasing manner (i.e. graphs, text boxes, photos). They specifically expressed interest in knowing if ICBT reduced the need for future face-to-face care and resulted in cost-savings.

Quantitative findings

Of the 14 items assessing implementation processes, the vast majority suggested that participants agreed that the OTU had a positive impact on ICBT implementation in most regards (e.g., developing a usable website for ICBT delivery, providing technical and clinical assistance, developing polices, securing funding, reporting on patient utilization and outcomes, training therapists, screening patients for ICBT, matching patients to therapists, treating additional patients when health region therapists were not available). Lower ratings, however, were obtained on the extent to which ICBT was planned in advance and on informing all staff in the health region about ICBT.


There is growing research demonstrating ICBT is effective for treating depression and anxiety and overcomes numerous barriers to accessing treatment [18]. Nevertheless, use of ICBT in clinical practice settings remains limited. This study fills a considerable gap in the literature by describing an external facilitation model that was used to implement ICBT within dispersed community mental health clinics in one Canadian province. We also report on contextual facilitators and barriers to ICBT implementation in order to understand factors that contributed to ICBT implementation and those that could impact future adoption and use of ICBT. Detailing our implementation strategies, as well as barriers and facilitators, may assist managers and therapists in other clinical settings who aspire to implement ICBT using a similar model.

Facilitators and barriers

Intervention characteristics and implementation processes appeared to have the greatest positive impact on ICBT implementation. In terms of the intervention, the relative advantages, design quality, and strength of evidence of the ICBT program appeared to be important factors facilitating ICBT implementation. Examination of participants’ responses to rating scales revealed other aspects of the intervention that also likely facilitated ICBT implementation including the transdiagnostic nature of the program, the brevity and effectiveness of the program, and low cost and external development of the program. Processes used to implement ICBT also served to facilitate ICBT implementation, especially in terms of engaging diverse stakeholders in ICBT implementation (e.g., patients, therapists). Participants also reported a positive perception of the diverse actions taken by the external facilitation Unit to implement ICBT in the clinics (e.g., managing the website, providing education and technical and clinical assistance, securing external funding). In comparison, the outer setting, inner setting, and individual characteristics were less consistently regarded as having facilitated ICBT implementation.

The inner setting emerged as representing the greatest barrier to ICBT implementation especially in terms of lack of readiness for implementation and implementation climate. Specifically, ICBT implementation was considered to be negatively impacted by limited resources being allocated to ICBT and perceptions that ICBT was of lower relative priority compared to face-to-face therapy. Of note, while therapists, on average, treated only 11 patients, this was still deemed to be taxing and difficult to integrate with existing patient workloads. Participants also felt inadequate attention was given to setting goals, reflecting and addressing challenges, and providing incentives and rewards to therapists; these factors were considered to hinder implementation.

While the inner setting clearly emerged as the greatest barrier to ICBT implementation, analyses suggested that there were some additional barriers to implementation. Low adaptability of ICBT in terms of support and length of treatment was perceived as a barrier to implementation. Participants also indicated that ICBT implementation would have been greater if additional clinical concerns were addressed with ICBT (e.g., alcohol or drug problems). Other barriers to implementation included negative beliefs about ICBT among some therapists and the need for even greater engagement of stakeholders in ICBT.

The results of the current study show consistency with a recent systematic review on factors that influence implementation of e-health interventions [10]. Consistent with this recent review, we found that ICBT implementation amongst mental health clinics was impacted by diverse constructs and no single construct explained implementation. Furthermore, similar to this review, we identified that the inner setting implementation climate negatively impacted ICBT implementation and implementation processes, such as planning, engaging, reflecting and evaluating, positively impacted implementation. Unique from this review, in the current study, we found that ICBT implementation was significantly facilitated by the relative advantages of ICBT, the quality of the ICBT program, and evidence supporting ICBT. These variables did not emerge as prominently in the review of e-health implementation; this could reflect that Ross et al. examined many forms of e-health (e.g., electronic medical records, decision support) and that intervention characteristics play a more prominent role when it comes to implementation of therapeutic programs. Evidence supporting ICBT may have been a particularly strong facilitator in our study as the implementation approach involved an initial workshop that incorporated evidence supporting ICBT. Furthermore, on a regular basis, we provided clinics with information on outcomes of ICBT and also regularly posted results of studies on ICBT on the Unit website. A further discrepancy is that in the past review, external policies and incentives were identified as highly beneficial for e-health implementation. This variable did not emerge in the current study, but could reflect that participants in our study were therapists and managers as compared to those involved in overseeing health policy. Of interest, in the current study relative priority of e-health within the inner setting emerged as a prominent barrier to implementation; Ross et al., [10] identified no past research on this construct suggesting the present study makes a unique contribution to the e-health implementation research.

Overall, we found CFIR to be a useful framework for guiding and examining multiple contextual factors impacting ICBT implementation. Similar to others, the CFIR terminology and language were coherent [12]. We did not find any need to add categories to the CFIR framework. In the future, it would be valuable to explore whether the variables that were identified as impacting ICBT implementation in the current context also explain implementation in other clinical settings. In this study, we used a bespoke survey that may prove helpful to others who are interested in evaluating implementation with respect to CFIR. This format had the advantage of being cost-effective for capturing information across multiple sites.

Study strengths and limitations

It is critical to study ICBT from an implementation perspective given that barriers and facilitators to implementation can impact the utility and sustainability of ICBT. Most past research on ICBT has focussed on ICBT within specialized clinics; however, it is important to consider implementation within settings where it may be more challenging to offer ICBT given multiple demands on therapist time. This is the first multi-site evaluation of the factors impacting uptake of ICBT in community mental health clinics. Lessons learned from implementation studies assist with local implementation, but also support widespread adoption of an intervention, allowing others to consider the strengths and challenges of using the facilitation model to implement ICBT, and to anticipate specific barriers and facilitators to ICBT implementation [12].

An additional strength of this study is that we utilized the CFIR to understand implementation barriers and facilitators; this is a comprehensive model built on the review of many other implementation theories [12]. We used both open-ended questions and rating scales to explore implementation processes and information was collected from two perspectives (managers and therapists). While there was convergence between methods of data collection, the ratings scales appeared most useful for identifying facilitators of ICBT implementation, while the open-ended responses were helpful for understanding barriers and potential areas of improvement. A final strength is the high participation rate.

In terms of limitations, it should be noted that qualitative analyses have the potential to be subjective and influenced by coder bias. In order to offset this bias, however, multiple coders were used and quantitative ratings supplemented the qualitative findings. A further limitation of this study is the sample size which could limit generalizability of the findings. The sample size also limited our ability to examine differences in implementation experiences between managers and therapists and uncover differences between clinics with higher and lower ICBT implementation. Also of note, there was a high degree of staff turnover and 17 therapists who provided ICBT had either resigned or retired prior to conducting the survey and therefore did not participate. It should be noted that facilitators and barriers to ICBT implementation are likely linked to the model of implementation, the local population, and the health care system. For example, with different implementation models, there may be less emphasis placed on sharing evidence regarding the intervention and this factor may impact implementation in a different fashion than what was found in our study. As another example, in a different region, funded by private insurance or with better access to mental health care, different facilitators and barriers to ICBT implementation may emerge. Future research is needed to examine ICBT implementation in different settings, with different populations, using different models. The open-ended questions and rating scales were developed specifically for this study. It is possible that findings may have varied if we used a different framework [19], formulated different questions, or utilized a different method of study (e.g., focus groups). Given our interest in examining facilitators and barriers to ICBT implementation, all participants in this study were directly involved in ICBT implementation. Perceptions of ICBT were highly favorable among this group and could be quite different among providers without experience with ICBT. It should also be noted that we assessed barriers and facilitators retrospectively after participants had experience with ICBT. Given the retrospective observational methodology, it is difficult to fully disentangle which facilitators and barriers were most important.

Future directions

There are a number of strategies that could potentially improve ICBT implementation in the future that are important for other groups to consider who have an interest in using a similar facilitation model to implement ICBT. In terms of the intervention, it would be beneficial to explore if implementation could be improved if the ICBT program was adapted to meet the needs of the patients, especially in terms of program length and amount of therapist support. Rather than offering standardized ICBT it may be valuable to explore if reach, outcomes, and implementation of ICBT could be improved if ICBT were personalized to the patient. This would be consistent with the notion of “patient-centered care”, which refers to “providing care that is respectful of and responsive to individual patient preferences, needs, and values” , p. 6 [20]. There is considerable research demonstrating that patient-centered care is associated with greater patient satisfaction and self-management [21]. Although it seems intuitive that personalized ICBT would improve adherence and outcomes, it is important to examine this systematically. Another possibility is that personalization may contribute to non-therapeutic discussion and drift away from the key tasks of ICBT, ultimately resulting in a reduction in patient adherence and outcomes.

In terms of the inner setting, it would be valuable to systematically examine if uptake could be improved if adjustments were made to resources dedicated to ICBT. Some participants suggested that ICBT should only be offered by a centralized unit while others suggested that settings other than mental health clinics should deliver ICBT (e.g., primary health care clinics). Others felt that within the clinics it would be desirable to have either fewer therapists or alternatively more therapists deliver ICBT. Still others felt that therapists should be given more time to offer ICBT. Of interest, within the OTU, consistent with how ICBT is described in the literature [18], most therapists need 15 to 20 min, per patient, per week to provide support. This reflects that ICBT materials reduce the need for therapists to provide information to patients. A number of therapists in the clinics, however, reported taking 45 min or more to provide support. This could reflect lack of experience or that therapists are having difficulties with the model of care and attempting to provide the same level of support that they provide in face-to-face care. It is possible that spending additional time on addressing inner setting barriers to ICBT implementation, such as workflow, could have resulted in greater use of ICBT beyond current levels. Past research has shown, for example, that compatibility of e-health programs with workflow positively facilitates e-health implementation [10]. In general, in the current trial, no specific goals were set for implementation, internal champions were not systematically identified, and incentives for delivering ICBT were not used; these approaches could represent additional strategies that could be examined to increase uptake in the future. It is also quite possible that addressing beliefs about ICBT, such as beliefs that face-to-face care is of greater priority than ICBT, could also serve to improve ICBT uptake in the future.

Another suggestion for improving ICBT implementation would be to explore if greater time and resources for engagement would improve uptake and use of ICBT. Among participants, there was a call to systematically examine ICBT as a first step in care prior to being offered face-to-face services. Currently, ~15% of the patients who receive ICBT report they are on waiting list for face-to-face services. It would be valuable to systematically examine whether need for face-to-face care is reduced if patients receive ICBT while waiting for face-to-face care. A previous study did not find offering ICBT to patients who were waiting for face-to-face treatment reduced the need for subsequent face-to-face care [22]. This finding requires replication, however, as the research was observational in nature. In terms of other directions, many participants, in the current study, requested more information about ICBT utilization and outcomes presented in a more engaging format. It would be valuable to examine whether providing this information would improve uptake and use of ICBT beyond current levels. Lastly, it would be valuable to examine the generalizability of the current findings to other regions in Canada or across the globe.


The study contributes to understanding factors that influenced ICBT implementation within community mental health clinics and serves to identify potential strategies for improving ICBT implementation in the future. The paper also presents a survey approach, including open-ended questions and rating scales, for capturing contextual factors that underlie implementation of programs. The paper adds to growing information available on CFIR.



Consolidated Framework for Implementation Research


Internet-delivered cognitive behavior therapy


Online Therapy Unit


  1. Kessler RC, Berglund P, Demler O, Jin R, Walters EE. Lifetime prevalence and age-of-onset distributions' of DSM-IV disorders in the national comorbidity survey replication. Arch Gen Psychiat. 2005;62(6):593–602.

    Article  PubMed  Google Scholar 

  2. Andrade LH, Alonso J, Mneimneh Z, Wells JE, Al-Hamzawi A, Borges G, Bromet E, Bruffaerts R, de Girolamo G, de Graaf R, et al. Barriers to mental health treatment: results from the WHO world mental health surveys. Psychol Med. 2014;44(6):1303–17.

    Article  CAS  PubMed  Google Scholar 

  3. Hadjistavropoulos HD, Alberts NM, Nugent M, Marchildon G. Improving access to psychological services through therapist-assisted, internet-delivered cognitive behaviour therapy. Can Psychol. 2014;55(4):303–11.

    Article  Google Scholar 

  4. Andersson G, Cuijpers P, Carlbring P, Riper H, Hedman E. Guided internet-based vs. face-to-face cognitive behavior therapy for psychiatric and somatic disorders: a systematic review and meta-analysis. World Psychiatry. 2014;13(3):288–95.

    Article  PubMed  PubMed Central  Google Scholar 

  5. Andersson G, Hedman E. Effectiveness of guided internet-based cognitive behavior therapy in regular clinical settings. Verhaltenstherapie. 2013;23(3):140–8.

    Article  Google Scholar 

  6. Mental Health Commission of Canada: E-mental health in Canada: Transforming the mental health system using technology. In. Ottawa, ON: Mental Health Commission of Canada; 2014.

  7. Hadjistavropoulos HD, Nugent MN, Alberts NM, Staples LG, Dear BF, Titov N. Transdiagnostic internet-delivered cognitive behaviour therapy in Canada: an open trial comparing results of a specialized online clinic and nonspecialized community clinics. J Anxiety Dis. 2016;42:19–29.

    Article  CAS  Google Scholar 

  8. Soucy JN, Owens VAM, Hadjistavropoulos HD, Dirkse DA, Dear BF. Educating patients about internet-delivered cognitive behavior therapy: perceptions among treatment seekers and non-treatment seekers before and after viewing an educational video. Internet Interv. 2016;6:57–63.

    Article  Google Scholar 

  9. Musiat P, Goldstone P, Tarrier N. Understanding the acceptability of e-mental health--attitudes and expectations towards computerised self-help treatments for mental health problems. BMC Psychiatry. 2014;14:109.

    Article  PubMed  PubMed Central  Google Scholar 

  10. Ross J, Stevenson F, Lau R, Murray E. Factors that influence the implementation of e-health: a systematic review of systematic reviews (an update). Implement Sci. 2016;11(1):146.

    Article  PubMed  PubMed Central  Google Scholar 

  11. Harvey G, Loftus-Hills A, Rycroft-Malone J, Titchen A, Kitson A, McCormack B, Seers K. Getting evidence into practice: the role and function of facilitation. J Adv Nurs. 2002;37(6):577–88.

    Article  PubMed  Google Scholar 

  12. Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci. 2009;4:50.

    Article  PubMed  PubMed Central  Google Scholar 

  13. Brehaut JC, Eva KW. Building theories of knowledge translation interventions: use the entire menu of constructs. Implement Sci. 2012;7:114.

    Article  PubMed  PubMed Central  Google Scholar 

  14. Saskatchewan Bureau of Statistics: Saskatchewan Population Report: 2016 Census of Canada. In. Regina, Saskatchewan: Government of Saskachewan; 2016.

  15. Marchildon GP, Hadjistavropoulos HD, Koocher GP. Health psychology wtihin the health-care system. In: Hadjistavropoulos T, Hadjistavropoulos HD, editors. Fundamentals of health psychology. Don Mills, Ontario: Oxford University Press Canada; 2014.

    Google Scholar 

  16. Sunderland A, Findlay LC. Perceived need for mental health care in Canada: results from the 2012 Canadian community health survey-mental health. Health Rep. 2013;24(9):3–9.

    PubMed  Google Scholar 

  17. Hsieh HF, Shannon SE. Three approaches to qualitative content analysis. Qual Health Res. 2005;15(9):1277–88.

    Article  PubMed  Google Scholar 

  18. Andersson G. Internet-delivered psychological treatments. Annu Rev Clin Psychol. 2016;12:157–79.

    Article  PubMed  Google Scholar 

  19. May C. Towards a general theory of implementation. Implement Sci. 2013;8:18.

    Article  PubMed  PubMed Central  Google Scholar 

  20. Institute of Medicine. Crossing the quality chasm: a new health system for 21st century. Washington, DC: National Academies Press; 2001.

    Google Scholar 

  21. Rathert C, Wyrwich MD, Boren SA. Patient-centered care and outcomes: a systematic review of the literature. Med Care Res Rev. 2013;70(4):351–79.

    Article  PubMed  Google Scholar 

  22. Kenter R, Warmerdam L, Brouwer-Dudokdewit C, Cuijpers P, van Straten A. Guided online treatment in routine mental health care: an observational study on uptake, drop-out and effects. BMC Psychiatry. 2013;13:43.

    Article  PubMed  PubMed Central  Google Scholar 

Download references


The authors would like to acknowledge research staff, therapists and patients associated with the Online Therapy Unit, University of Regina, as well as staff, therapists and patients from the community mental health clinics located within the Regina Qu’Appelle, Saskatoon, Five Hills, Sun Country, Prince Albert Parkland, Kelsey Trail and Cypress Health Regions The support of Saskatchewan Health in facilitating the project is gratefully acknowledged. We would also like to acknowledge Luke Schneider for his review of the survey used in this study. We are also grateful to Professor Nikolai Titov and Dr. Blake Dear from Macquarie University for providing access to the ICBT Wellbeing Course.


This work was supported by funding provided by the Canadian Institutes of Health Research (reference number 293379) with matching funding provided by the Saskatchewan Health Research Foundation and the Rx & D Health Research Foundation. Funders had no involvement in the study design, collection, analysis, or interpretation of the data.

Availability of data and materials

The materials and datasets used in the current study are available from the corresponding author on reasonable request.

Author information

Authors and Affiliations



HH conceived the project, designed the study and survey, analyzed the data, and led the writing of the manuscript. MN co-designed the survey, co-analyzed the data, and read and commented on the manuscript. DD was involved in analysis of the data and writing of the manuscript. NP was involved in writing the manuscript. All authors read and approved the final manuscript.

Corresponding author

Correspondence to H.D. Hadjistavropoulos.

Ethics declarations

Ethics approval and consent to participate

All participants consented to participate. The study was approved by the University of Regina Research Ethics Board (#31R1314).

Consent for publication

Not applicable.

Competing interests

The authors declare that they have no competing interests.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Additional file

Additional file 1:

Open-ended and closed-ended survey assessing CFIR constructs. (PDF 114 kb)

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (, which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver ( applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Hadjistavropoulos, H., Nugent, M., Dirkse, D. et al. Implementation of internet-delivered cognitive behavior therapy within community mental health clinics: a process evaluation using the consolidated framework for implementation research. BMC Psychiatry 17, 331 (2017).

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: