Skip to main content

Implementing internet- and tele-based interventions to prevent mental health disorders in farmers, foresters and gardeners (ImplementIT): study protocol for the multi-level evaluation of a nationwide project



Farmers are a vulnerable population for developing depression or other mental health disorders due to a variety of risk factors in their work context. Beyond face-to-face resources, preventive internet- and tele-based interventions could extend available treatment options to overcome barriers to care. The German Social Insurance Company for Agriculture, Forestry and Horticulture (SVLFG) implements several guided internet- and mobile-based interventions and personalised tele-based coaching for this specific target group provided by external companies within a nation-wide prevention project for their insured members. The current study aims to evaluate the implementation process and to identify determinants of successful implementation on various individual and organisational levels.


The current study includes two groups of participants: 1) insured persons with an observable need for prevention services, and 2) staff-participants who are involved in the implementation process. The Reach, Effectiveness, Adoption, Implementation and Maintenance (RE-AIM) framework and the Consolidated Framework for Implementation Research (CFIR) will be used to track and evaluate the implementation process. A mixed-method approach will provide insights on individual and organizational level (e.g. degree of normalization, readiness for change) and helps to identify determinants of successful implementation. In-depth insights on experiences of the participants (e.g. acceptance, satisfaction, barriers and facilitating factors of intervention use) will be yielded through qualitative interviews. Focus groups with field workers provide insights into barriers and facilitators perceived during their consultations. Furthermore, intervention as well as implementation costs will be evaluated. According to the stepwise, national rollout, data collection will occur at baseline and continuously across 24 months.


The results will show to what extent the implementation of the internet- and tele-based services as a preventive offer will be accepted by the participants and involved employees and which critical implementation aspects will occur within the process. If the implementation of the internet- and tele-based services succeeds, these services may be feasible in the long-term.

Trial registration

German Clinical Trial Registration: DRKS00017078. Registered on 18.04.2019.

Peer Review reports


Depressive disorder is a common condition with a 12-month prevalence rate of 8.56% in European countries [1]. Previous research has shown that farmers [2,3,4,5] and employees in related occupations (e.g. forestry and fishery workers) [6] are at an increased risk for developing mental health problems, especially depression, compared to non-agricultural occupations. Risk factors for mental disorders associated with agricultural jobs include: work-related stress [7, 8], longer work hours [3], poor harvest and weather [2], financial problems [9], health problems [8, 10] and exposition to pesticides [8].

At the same time, farmers mainly live in rural areas with limited mental health care. In Germany, the availability of on-site psychotherapy is additionally restricted by long waiting times in rural areas [11, 12]. Furthermore, stigma surrounding mental disorders and resistance to mental health care are reported in rural areas [5]. Especially male farmers are often constrained to friends and family for mental health support, while professional face-to-face therapeutic treatments are rarely obtained [5]. Thus, alternative ways to deliver treatment might be beneficial for farmers [2].

Existing evidence-based treatments for mental disorders have shown to only reduce the burden of mental health disorders by a third [13]; however, symptoms of mental disorders can be effectively reduced with preventive interventions [14, 15]. As a recent meta-analysis showed, psychological interventions can prevent or delay the onset of Major Depressive Disorder (MDD) by focusing on individuals with increased risk (selective prevention) or with subclinical depressive symptoms (indicated prevention) [16].

Internet- and tele-based interventions

Easily accessible preventive interventions, as implemented in this study, could increase farmers’ ability to overcome reported challenges. Internet- and mobile-based psychological interventions (IMIs) have been shown to be effective in working participants [17] as well as for preventing MDD onset and treating subthreshold depression in the general population [16, 18,19,20,21]. In a similar context, researchers have begun to evaluate online interventions in Australian farmers to reduce stigma surrounding suicide and to combat rural isolation for farmers [22].

Although access to the internet has increased significantly in Germany in recent years, rural areas still face problems using Internet services [23]. Therefore, tele-based coaching is an important addition to existing preventive programmes against depression, especially in rural communities. Evidence also suggests that tele-based coaching, in comparison to treatment-as-usual, is effective in reducing symptoms of depression [24].

Despite its effectiveness, the uptake of health care interventions in routine practice remains low and costly [25, 26]. Only 14% of evidence-based health care interventions are used in routine care [27]. While many studies focus on the implementation of interventions for the treatment of mental disorders, there is little research on the implementation of preventive interventions. The few studies relate to the prevention of eating disorders and depression in adolescents and young adults [28, 29].

Implementation in rural areas

Technology-based solutions appear to be an increasingly viable option as they have the potential to overcome disparities in access for rural mental health care services [30, 31]. In addition, computerised Cognitive Behavioural Therapy interventions for anxiety and/or depression seem to be acceptable among people living in rural and remote areas [32]. Moreover, barriers and facilitating factors for the implementation of health care interventions in rural areas have been studied: Current evidence suggests that close collaborations with professional institutions, training of employees and perceived credibility of the treatment may facilitate successful interventions [33]. These facilitating factors may be achieved by supporting government agencies, qualified program managers or empirical evidence of efficacy [33]. Furthermore, increased public awareness for mental disorders [34] as well as for the benefits of telehealth [35] might be beneficial.

In addition to the participants, perspectives of all people involved in the implementation and delivery of this treatment should be considered. For example, health care staff in rural areas may prioritise staff attitudes and beliefs towards treatment [36, 37] as well as the leadership and organisational structures of the clinic and the community [37]. Likewise, organisational readiness within rural communities is imperative for the successful implementation of telehealth services [38].

Implementing mental health services in rural areas faces specific difficulties since existing studies disregard the diversity of rural settings, are rarely based on well-developed theories and show methodological weaknesses [39]. Studies exist regarding particular aspects of implementation in rural areas (e.g. acceptance of interventions). However, there is a lack of comprehensive, multi-level implementation research in routine care, especially with regards to the implementation of technology-based interventions for mental disorders in rural areas.

A nationwide pilot project

This study is part of a national depression prevention program for farmers, gardeners and foresters (“With us in balance”) carried out by the German Social Insurance company for Agriculture, Forestry and Horticulture (SVLFG, The study aims to evaluate the implementation of tailored IMIs and a personalised tele-based coaching as an extension to existing face-to-face group interventions. In addition to the implementation study, it is evaluated in parallel running, randomised controlled trials whether the internet- and tele-based interventions are (cost-)effective in reducing depressive symptoms and other mental health concerns as well as in preventing the onset of clinical depression in this specific target group of farmers, gardeners and foresters [40, 41].Footnote 1

Evaluation frameworks

Implementation success is considered as a multi-dimensional construct. In this study, the evaluation is based on the Reach, Effectiveness, Adoption, Implementation and Maintenance (RE-AIM) framework [42] as it helps to structure this multifaceted evaluation on individual and organisational levels. The RE-AIM framework is one of the most used evaluation frameworks in the implementation setting and facilitates to assess the quality, speed and impact of implementation efforts. In addition, the use of the Consolidated Framework for Implementation Research (CFIR) enables a deeper understanding of the mentioned ‘implementation’ dimension, as CFIR offers a pragmatic structure for approaching compound and interacting states in the implementation of interventions [43].

Staff are part of a dynamic interaction within the organisation and are a key component to successful implementation. Previous literature showed the importance to examine different levels, when implementing in rural areas, e.g. staff attitudes and beliefs [36, 37], management and organisational structures, and organisational readiness [38]. The following constructs focusing on the involved employees aim to capture different aspects of the implementation process:

  • Degree of Normalisation: Normalisation refers to activities that people pursue to integrate an innovation into their everyday routine. According to the Normalisation Process Theory, normalisation is defined as the extent to which a new intervention is considered as a normal part of daily work [44].

  • Determinants in the implementation process: The Theoretical Domains Framework (TDF) provides an integrative framework model to identify drivers of behavioural change that could influence determinants in the implementation process [45].

  • Organisational leadership: Minimal attention is given in research to the organisational context, although it may have a significant impact on the implementation of evidence-based practices [46]. Therefore, this study collects information regarding implementation leadership which may have an impact on the program’s ability to drive change and innovation [47].

  • Organisational readiness for change: This construct refers to the extent to which employees of an organisation are willing to implement organisational change, with particular focus on psychological and behavioural aspects [48].

Research questions

The aim of this study is the multi-level evaluation of the implementation success and stepwise nationwide rollout of the following new preventive services: a) GET.ON online health trainings with 7 guided IMIs, and b) personalised tele-based coaching provided by external companies in addition to existing on-site group interventions offered by the SVLFG.

Our study seeks to address the following research questions:

  • How successful is the implementation of internet- and tele-based interventions in terms of reach, effectiveness, adaptation, implementation and maintenance on an individual (participants, staff) and organisational level?

  • How effective is the implementation in terms of intervention uptake, normalisation into everyday life, and implementation costs?

  • What are the experiences and impressions of the participants receiving internet and tele-based interventions? In particular, to what degree do participants report: a) acceptance and use of the technology-based treatment, b) satisfaction with the intervention considering their expectations, and c) perceived barriers and facilitators to use the intervention?

  • Which barriers and facilitating factors can be identified for advising on prevention services at staff level?

  • What determinants for implementation success in rural areas can be identified by focusing on employees involved in the referral process?

  • Which similarities and differences can be identified between internet- and tele-based interventions with regards to implementation success and other determinants metrics of implementation effectiveness?


Design of the study

This study is a prospective, longitudinal implementation study and is described according to the Standards for Reporting Implementation Studies (StaRI) checklist [49], see Supplementary material. A mixed-methods approach using qualitative interviews, focus groups, quantitative surveys and reporting data is utilised for a comprehensive understanding of the implementation process. The evaluation is based on the RE-AIM framework [42] and the CFIR [43].

The implementation follows a stepwise, nationwide rollout plan which determines the availability of the internet- and tele-based interventions and the enrolment of the insured persons in each rollout region. Each region (2–7 federal states of Germany) undergoes these implementation phases at different times, starting with a pilot implementation phase in Bavaria and Schleswig-Holstein for 1.5 years and following with three rollouts (A, B and C) over 2 years. The initial pilot implementation phase was extended for various reasons (e.g. additional time needed to implement internal structures at SVLFG such as the referral process and documentation of consultations with insurers, additional time to conduct trainings, and unforeseen delayed availability of the tele-based coaching). Based on the Conceptual Model of Implementation Phases [50], the study can be divided into the following phases: exploration, preparation, implementation (6–12 months) and maintenance (12–24 months). The duration of the phases was defined in accordance to the RE-AIM framework that recommends 6 to 12 months for the implementation phase and at least 2 years for the maintenance phase [42]. In addition, the enrolment plan of the SVLFG stipulates that the rollout area is expanded every 6 months and focuses on implementation in this region within these 6 months Fig. 1.

Fig. 1

Project timeline depicting the Conceptual Model of Implementation Phases [50] including exploration (E), preparation (P), active implementation (I) and sustainment (S) adapted to the stepwise rollout plan


The insured individuals are advised on the preventive services by approximately 350 employed field workers and in-house staff of SVLFG during their consultations via phone or on-site. In addition, the prevention services are advertised widely through PR activities of the SVLFG (e.g. newspaper articles, radio reports, SVLFG member magazine, lectures and information events) in the federal states where the interventions have been implemented. Interested individuals receive detailed information regarding the offered prevention services and their eligibility at the SVLFG central call centre. Eligible insured persons can choose whether they prefer single (e.g. internet- or tele-based interventions) or group preventive services. The group interventions are delivered on-site and include various topics on mental health issues for this target group (e.g. dealing with stress, handing over the business to the next generation). The registration process is conducted by SVLFG.

Data collection starts with rollout A in April 2019 and is expected to be finished at the project end (April 2021).


The internet- and tele-based interventions are described according to the Template for intervention description and replication (TIDieR) [51], see Supplementary material. They are briefly outlined below.

Tailored IMIs

Seven IMIs are provided by the GET.ON Institute ( Their effectiveness in the prevention and treatment of mental disorders has been demonstrated in more than 30 randomised controlled trials [19, 20, 52,53,54,55,56,57,58,59,60,61,62,63,64,65]. The IMIs are individually tailored to the participants based on their symptoms, risk profile and needs, and follow the selective prevention approach (e.g. providing trainings for a group with increased risk for mental disorders). The different IMIs are addressing risk factors for the incidence of depression including subclinical depressive symptoms [66], insomnia [67], stress [68], anxiety [69], chronic pain [70], harmful alcohol use [71, 72] and subclinical depressive symptoms in the context of diabetes [73]. The IMIs are adapted to the target group of farmers, foresters and gardeners in terms of content (e.g. descriptions of mock individuals facing similar struggles) and graphics to meet the needs of the participants. Variation of content and duration of the seven IMIs is illustrated in Fig. 2.

Fig. 2

Online content, guidance and duration of the GET.ON online health trainings

To begin the intervention phase, the participant completes a computer-adaptive psycho-diagnostic assessment which is based on self-report questionnaires assessing mental health issues. The type of online training is chosen during the initial interview with the e-coach depending on the participant’s symptom profile, needs and preferences. The modules include psychoeducation, testimonials of mock individuals, in-depth exercises, a diary and homework to improve mental health and wellbeing. Depending on the training type, participants have access to additional modules. After each module, the participant receives individual feedback by email or by phone from the e-coach which helps to improve adherence of e-health interventions [74].

The e-coaches are psychologists with a university degree who have at least started a psychotherapeutic training. The training of the e-coaches consists of supervision by licensed psychotherapists in CBT training and information about IT-matters. A detailed description is published in the study protocols elsewhere [40, 75,76,77,78,79,80].

Personalised tele-based coaching

The personalised tele-based coaching is provided by the company IVPNetworks ( and focuses on general mental health problems. The coaches are psychologists with trainings either in cognitive behavioural therapy, systemic therapy, psychodynamic psychology, hypnotherapy or other therapeutic trainings. Licensed psychotherapists are responsible for supervision. The coaches support participants in recognising and understanding conflict patterns in order to effectively cope with issues by activating their own resources. This is pursued by approaching the participant’s personal situation and stressors (e.g., financial burden, family problems, work-related stress).

Participants are registered with their contact information in the IVPNetworks management and documentation platform (IVPnet 2.0). A case manager at IVPNetworks assigns the participant to a coach. The coaching is personalised in terms of amount, frequency and duration of the sessions, depending on the individual participant’s needs. Personalised coaching means that there are no standardised manuals or fixed procedures for the coaching. Different therapeutic methods are used depending on the therapeutic background of the coach (which will be monitored in the study). If indicated, participants are supported in finding on-site social and health care services to complement the tele-based coaching (e.g., socioeconomic consultants, agricultural family counselling). Alternatively, on-site coaching can be arranged if a participant no longer prefers tele-based coaching. Common content and duration of the personalised tele-based coaching at IVPNetworks are illustrated in Fig. 3.

Fig. 3

Content and duration of the personalised tele-based coaching at IVPNetworks

In addition to the personalised tele-based coaching, IVPNetworks offers a crisis hotline that can be used 24/7 by those insured at SVLFG. The crisis hotline is managed by trained psychologists and is a resource for acute psychological emergencies and crisis situations. As part of the crisis hotline, up to five follow-up contacts are possible.


The study design includes two groups of participants involved in the implementation process:

Group 1: insured persons at SVLFG

Inclusion criteria for insured persons at SVLFG includes: being an insured member of the health insurance scheme OR the pension fund of the SVLFG, being an agricultural contractor OR a collaborating spouse OR a family member or a pensioner, being over 18 years old, having access to the internet and/or a telephone, and providing informed consent.

Persons will be excluded if they currently suffer from a mental disorder and need extensive therapeutic service. Individuals may be excluded during the consultation in the call centre (if the interested insured person reports a mental disorder), during the assessment by the service provider or during their participation in case of a suspected mental disorder. If there is a suspicion, the external service provider advises the participant that a consultation with a general practitioner is recommended. The insured individual gets support in the search for further treatment options. In addition, people can be excluded for the tele-based intervention, if they have substantial hearing impairment or for the online interventions, if they have a visual impairment.

There are 465,000 eligible insured members of SVLFG nationwide, and the sample size is not capped. The number of participants enrolled in this study depends on the rollout stage of the implementation. Eligible insured persons can choose the specific intervention based on their preference and needs which reflects the conditions of routine care. A sample size of N = 100 per intervention is desirable to achieve reliable results. Based on previous experiences in the pilot phase of the implementation in two federal states, we expect a sample size of approximately 500 to 800 participants for the national implementation area.

Group 2: staff-participants at SVLFG, GET.ON and IVPNetworks

Field worker, in-house staff and call centre agents at the implementing company SVLFG

Approximately 350 field workers visit the insured members on-site. In-house staff counsel the insured persons by telephone. Both field workers and staff advise insured members on technical prevention, rehabilitation, etc. and additionally inform and advise members regarding the internet- and tele-based prevention services. They also forward interested persons to a nationwide call centre. Depending on the rollout phase (which includes a training on all health services offered by the SVLFG), the field workers begin advising members on the internet- and tele-based interventions at different times. Twenty-five call centre agents advise the insured individuals on various health offers including the newly implemented internet- and tele-based interventions. Additionally, call centre agents check the insurance eligibility criteria and are responsible for the enrolment in the prevention services.

Thus, implementation of internet- and tele-based interventions requires that these employees undergo changes to their work environment, especially at the behavioural level, by adapting their previous working routines to the new requirements.

Implementation team of SVLFG

Members of the implementation team are responsible for the planning and realisation of the implementation activities. Employees who work in two or more roles (e.g. as a field worker, in-house staff and/or call centre agent) are excluded from the staff surveys for field workers, in-house staff and call centre agents to avoid conflation of the data, as such persons cannot perform both roles simultaneously without bias.

Coaches at GET.ON and IVPNetworks

Coaches are involved in the delivery of the internet- or tele-based interventions and work as psychologists or therapists on a freelance basis or employment contract for GET.ON or IVPNetworks.

Approximately 150 staff-participants at GET.ON, IVPNetworks and SVLFG are expected to participate in the study. All employees of the described staff-groups who sign the informed consent are able to participate in the study. There are no specific eligibility criteria. If additional professional groups become involved in the future course of the implementation of these services, it will be decided whether these employees will be included in the surveys.


The used measures are summarised in Table 1, followed by a description of each RE-AIM dimension and selected outcomes. Measurements include routine data, staff reviews of intervention consultations as well as usage of the interventions by insured individuals, online-assessments, qualitative interviews and focus groups.

Table 1 RE-AIM dimensions, measures and assessments


To assess reach, the number and characteristics of the potentially eligible insured individuals at SVLFG, as well as exclusion, referral and uptake rates of the internet-, tele-based and on-site group preventive services will be reported. The following uptake levels will be considered in order to illustrate different levels of engagement:

  • Referral rates: The absolute number of: a) referrals by the SVLFG field workers (operationalised as consultations about internet- and tele-based interventions), b) referrals by the call centre agents to the internet- and tele-based prevention services (IMIs: sending out a code for the registration at the online-platform, tele-based coaching: registration on the IVPnet platform), and c) the first contact with the external service provider (IMIs: first log in at online-platform to symptom screening; tele-based coaching: agreement for first tele-based coaching appointment).

  • Uptake rates of internet- and tele-based interventions:

    • Intervention not started: The absolute number of referred persons who completed the assessment phase (IMIs: psycho-diagnostic assessment, tele-based coaching: first coaching assessment session) after the first contact with the external service provider, but have not started the intervention.

    • In treatment: The absolute number of persons who have started the intervention after assessment phase (IMIs: first login into first module; tele-based coaching: receive tele-based coaching at minimum second session).

    • Treatment not completed: Absolute number of persons who completed less than 50% vs. 70% of the intervention (IMIs: completed less than 50% vs. 70% maximal online modules within 13 weeks, have not logged in for at least 8 weeks or have interrupted the training for other reasons; tele-based coaching: completed less than 50% vs. 70% of the total call minutes, did not answer for at least 8 weeks or cancelled the coaching for other reasons).

    • Treatment completed: a) In previous e-mental health studies, the completer criteria differed between 50 to 100% of completed units [81,82,83,84]. Research also indicates that a minimal treatment for subthreshold depression (e.g. six phone calls each with a maximum of 15 min) can also be effective in preventing MD [85]. Therefore, we defined two criteria with different treatment dosis as following: Absolute number of persons who completed at least 50% vs. 70% of online modules or call minutes. b) Clinical judgment by coaches at GET.ON Institute and IVPNetworks assessed with the Helping Alliance Questionnaire (HAQ) based on 11 items [86] in order to supplement the quantitative criteria for completion.

  • Uptake of other services: This may include the crisis hotline, a second guided IMI, diagnostic clarification at IVPNetworks (via telephone or on-site), referrals to social or health service centres (e.g. socioeconomic or family consultants) during the tele-based coaching.

In addition, the estimated number of people exposed to recruitment materials and the associated response rate will be assessed. Specific recruiting strategies are tested to examine successful ways of addressing the target population. To provide representative information, interested but not participating individuals, and enrolled participants will be compared based on routine data. Differences in reach between the internet-, tele-based and on-site preventive services will be investigated as well.


Effectiveness will be assessed at pre- and post-treatment (3 months after first log in on the GET.ON platform or 6 months after the registration at IVPnet) with online surveys consisting of frequently used and well validated self-report questionnaires. Depressive symptoms are assessed using the German Version of the Patient Health Questionnaire 9 (PHQ-9) [87] with high values on internal consistency (α = 0.88) [88]. Generalised Anxiety Disorder severity is measured by the Generalised Anxiety Disorder 7 item self-report scale (GAD-7; α = 0.89) [89, 90]. Perceived stress is assessed with the German Version of Perceived Stress Scale with 4 items (PSS-4; α = 0.84) [91]. Somatisation tendencies will be measured by using the Somatic Symptom Scale based on 8 items (SSS-8; α = 0.81) [92, 93]. The 12-item Assessment of Quality of Life instrument (AQoL–4D; α = 0.81) [94] will be used to assess health-related quality of life. Subjective capacity to work will be assessed by using the Subjective Prognostic Employment Scale (SPE; Guttman scaling: rep = 0.99) based on 3 items [95]. Side effects attributed to the intervention are collected in the post-assessment period by the Inventory for the Assessment of Negative Effects of Psychotherapy (INEP; 22 items; α = 0.86) [96] adapted to internet- and tele-based interventions.


Adoption will be assessed through the number and characteristics of the involved employees at SVLFG as well as at the external provider (GET.ON institute, IVPNetworks). Furthermore, metrics of the rollout regions and federal states, such as amount and characteristics of insured people and type of agricultural holdings will be described and compared to one another.


After a broad literature search, questionnaires were selected that fit the study context as best as possible (e.g. can also be used for employees who are involved in the referral and not in the service delivery itself).

Intervention characteristics

Potential adaptations to the existing interventions during the implementation process will be reported by the external service provider following the TIDieR checklist [51] halfway through the implementation and when concluding the project. Adaptations in the course of the implementation relate to referral and registration ways (e.g. to overcome barriers to participation). Furthermore, stakeholders’ (e.g. field workers, in-house staff, call centre agents) perception of perceived (dis-)advantages of internet- and tele-based interventions and the treatment credibility will be assessed using the Credibility Expectancy Questionnaire (CEQ) with 6 items [96]. The psychometric properties of the CEQ are characterized by high internal consistency (α = 0.84–0.85) and good values on test-retest reliability [97]. Intervention costs are recorded during the study as cost per participant for the respective intervention. Likewise, the use of the interventions by the insured individuals (including adherence, duration of intervention, number and duration of contacts, main topics, therapeutic background of coaches, etc.) will be monitored.

Outer setting

Constructs including participants’ needs and resources are related to the CFIR dimension ‘outer setting’ [43]. In assessing the outer setting, participant satisfaction is measured by using a German Version of the Client Satisfaction Questionnaire adapted for internet interventions (CSQ-I; α = 0.93) based on 8 items [65, 98]. The participants are asked about the usefulness and relevance of the interventions. Acceptance of the interventions is indirectly recorded via non-completer rates (indicator, see above). Approximately 20 semi-structured interviews each with participants from GET.ON chronic pain, from the six remaining IMIs and from the tele-based coaching will supplement the data collection and provide insights into the participants’ experience (e.g. acceptance, satisfaction, barriers and facilitators for intervention usage) with the internet- or tele-based interventions. The interview guide is based on the Unified Theory of Acceptance and Use of Technology (UTAUT) model [99] and the Discrepancy Theory of Satisfaction [100] with regards to the dimensions: technical quality of care, psychosocial quality of care, organisational conditions, spatial and technical equipment, treatment outcome, continuity of care and financing [100].

Inner setting

Constructs such as organisational culture and leadership engagement are related to the CFIR dimension ‘inner setting’ [43]. In assessing the inner setting, organisational readiness for implementing change for field workers, in-house staff, call centre agents and the implementation team at SVLFG is measured by using a German version of the Organisational Readiness for Implementing Change (ORIC) questionnaire with 12 items [101]. The ORIC questionnaire focuses on the degree to which employees are psychologically and behaviorally ready to implement organizational changes, which can be considered as a decisive factor for implementation success [101]. Good internal consistency (α = 0.88–0.92) is reported for the ORIC questionnaire [101]. Leadership of the implementation team as well as of the supervisors of field workers and call centre agents at SVLFG will be captured by using the Implementation Leadership Scale (ILS) based on 12 items [102]. The ILS can be seen as an efficient and short questionnaire which has been validated in settings of mental health with very high values on internal consistency (α = 0.93–0.97) [103].

Characteristics of the individuals

The CFIR dimension of ‘characteristics of the individuals’ encompasses knowledge and beliefs about the intervention, self-efficacy and other personal attributes of the staff participants at SVLFG [43]. The degree of normalisation among staff-participants (e.g. field workers, in-house staff, call centre agents) is assessed by the German translation of the NoMAD and 3 additional normalization questions; the questionnaire (20 items) refers to the normalisation process theory and assesses the extent to which the new implemented service is a normal part of the daily routine at work [104]. The NoMAD questionnaire is characterized by a high internal consistency (α = 0.89) and has already been validated in heterogeneous samples [105, 106]. Determinants of behavioural change (e.g. knowledge, belief, self-efficacy) are assessed with the Discriminant Content Validation of the TDF Questionnaire (DCV) with 32 items [107] among field workers, in-house staff and call centre agents. The DCV helps to understand factors that influence the behaviour in health care settings and can therefore makes it possible to improve the implementation of innovations. Additionally, about 20 focus groups using an interview guide based on TDF [45] with field workers are conducted to identify barriers and facilitating factors for advising on prevention services. The aim of this mixed-method approach is the optimisation of the implementation process or interventions based on the findings.

Implementation process

To capture the implementation process, the number and type of implementation and dissemination activities of SVLFG, GET.ON and IVP are reported every 3 months with an excel file (e.g. date, place, organisation, setting of the activity, target group, number of participants and content / aim of the activity). Implementation costs are reported every 3 months with an excel file and are defined according to the EU-project ImpleMentAllFootnote 2 project as the sum of personnel costs and other direct and indirect costs (indirect costs calculated at 20% of direct costs). In addition, the implementation team is interviewed every 6 months, particularly with regards to perceived effectiveness of the implementation in general and based on individual implementation strategies. These interviews also discuss barriers and facilitating factors in the implementation process and assessment of the current state of implementation. Quantitative (referral and uptake rates, barriers and facilitating factors in the implementation) and qualitative results (focus groups) about the progress and quality of implementation are prepared and delivered to the implementation team every 6 months to promote adaptions and improvements.


In assessing the maintenance of the implemented internet- and tele-based interventions, follow-up surveys with the insured participants will be conducted 12 months after first contact for the prevention project (e.g. 12 months after first login at the GET.ON platform or registration at IVPnet) to monitor the long-term impact of preventive interventions on mental health (see effectiveness measures, 60–65). Referrals to the SVLFG call centre are reported by the external service providers to capture all processes in the implementation that contribute to sustainable development. Uptake rates of the internet- and tele-based interventions are monitored in the maintenance phase of the respective rollout region. In terms of sustainability, we also record the use of a second IMI, the maintenance phase of the IMIs (e.g. monthly contacts for a year) or the prolongation of an IVP tele-based coaching and any other health care utilisation.

Normalisation is recorded in the maintenance phase using the questionnaire NoMAD [104] at the staff-level (e.g. field workers, in-house staff, call centre agents), and will be compared to the normalisation degree during the implementation phase. Additionally, the attitudes and experiences of the staff-participants involved in the implementation process are essential for sustained implementation. Therefore, a concluding survey with the staff-participants and important multipliers at SVLFG (e.g. management board) will be conducted at the end of the project to assess their professional view on the implementation and their intention to further use the internet- and tele-based prevention services. To ensure practical relevance, the item pool for this survey will go through an iterative feedback process with the initiators of the project.

Measuring implementation success

In line with the ImpleMentAll,Footnote 3 implementation success is defined as an increase in uptake of internet- and tele-based interventions (referral rates and levels of uptake, e.g. not started, in treatment, completer, non-completer), degree of normalisation among staff-participants (extent to which consultation to the internet- and tele-based interventions are perceived as a normal part of work) and efficiency outcomes (costs of implementation effort in relation to uptake and degree of normalisation).


Reporting of implementation results will follow the StaRI [49] statement. To assess the implementation success and to illustrate a broad picture of the implementation process in the context of rural areas we will combine qualitative and quantitative methods in a summative way. Quantitative and qualitative data are collected simultaneously to evaluate the implementation process. Qualitative data are used to provide a deep understanding, while quantitative data are used to provide breadth of understanding [108]. In addition, the results of the qualitative analysis are then validated in a quantitative survey. Qualitative surveys are only carried out on a small part of the sample, while all participants are invited to the respective quantitative surveys. The database is therefore larger for quantitative surveys. The analysis is based on guidelines for mixed-methods research defined by the NIH Office of Behavioural and Social Sciences Research [109]. In addition, the Consolidated criteria for reporting qualitative studies (COREQ) [110] are used.

Quantitative analysis

The evaluation of the implementation follows the RE-AIM dimensions as described above. Descriptive analyses (frequencies, means and percentages) will be performed to assess outcomes across all RE-AIM dimensions. We will evaluate the implementation success of the internet- and tele-based interventions separately.

To assess the representativeness of the participants compared to the target group, t-tests or non-parametric tests will be used. To assess intervention effectiveness, we analyse for different groups of participants (e.g. participants who have started treatment or completed treatment) by using independent sample t-test and chi-squared analyses. We will use t-tests or non-parametric tests to analyse differences between pre-, post- and follow-up-measurements. In order to counteract the issue of multiple comparisons, Bonferroni correction is used. The statistical significance level is set to p < .05. Within-group effect sizes and 95% CIs will be reported at all measurement points. Implementation effectiveness is assessed by comparing uptake rates, the degree of normalisation among staff-participants and efficiency outcomes (costs of implementation effort in relation to uptake and degree of normalisation) at different time points.Footnote 4

Furthermore, linear regression models are used to investigate differences between the interventions (guided IMIs, personalised tele-based coaching and on-site group workshops) regarding reach outcomes (referral and uptake rates) with intervention type as independent variable. By using further linear regression models we compare socio-demographic and clinical outcomes as dependent variables between the participants of the different prevention services (independent variable). Based on multiple linear regression models we analyse the influence of factors at staff level (credibility of the internet- and tele-based interventions, degree of normalisation, leadership support, organisational readiness to change and determinants of behavioural change) as independent variables on the number of referrals by field workers, in-house staff and call centre agents (dependent variables). Additionally, the implementation outcomes will be compared between the federal states and rollout areas at different phases of implementation.

According the intention-to-treat principle all observed data will be included. Patterns of missing data will be analysed and corrected. All analyses will be performed with R statistic software [111].

Qualitative analysis

Qualitative data is used for insights into the implementation process. Focus groups with field workers are conducted to understand barriers and facilitating factors of the consultation regarding the internet- and tele-based services. Interviews with insured persons enable insights in participants’ experience (e.g. satisfaction, acceptance). All qualitative data will be audiotaped and transcribed verbatim using MAXQDA. The analysis is based on the qualitative content analysis by Mayring [112]. To establish reliability, two independent raters will code the same transcripts using a coding guide and coding rules. The focus groups are evaluated inductively as well as deductively, based on the TDF [45] in two different coding passes. The interviews with the insured individuals are evaluated with an inductive-deductive approach.


This study provides novel and diverse information to the existing literature regarding the implementation of preventive treatments for depression. To our knowledge, this study is the first nationwide prevention project in Germany and the first comprehensive and multi-level implementation study in general that examines the implementation of internet- and tele-based interventions in farmers, gardeners and foresters in rural areas. The implementation process of different preventive services such as internet- and tele-based interventions will be compared. Insights into implementation of mental health services on an individual (participants, staff) and organisational level, within the context of rural areas, can be assessed. Additionally, the associated costs (implementation and intervention costs) will be considered. Furthermore, determinants for implementation success in rural areas can be identified by focusing on employees involved in the referral process. Based on these findings, recommendations for future interventions can be made.

The three parallel running RCTs [40, 41]Footnote 5 will deliver results on the clinical effectiveness of the internet and tele-based interventions under randomized controlled conditions. Results of the implementation study will provide information about the implementation of the internet- and tele-based interventions from a routine care perspective in terms of a hybrid study design simultaneously examining the impact of the interventions in routine care (e.g. pre-, post- and follow-up comparisons at participant level) and the implementation itself. This enables results based on two different, complementary study designs. The results will show to what extent the implementation of technologies as preventive offers will be accepted by the participants as well as employees involved in the implementation process. If the implementation of the internet- and tele-based services succeeds, these services may be feasible in the long-term and health care in rural areas could be sustainably improved.

Trial status

Recruitment has been started in April 2019. Participants will be continuously included until the end of the study. Data collection is ongoing.

Availability of data and materials

Results will be submitted for publication in a peer-reviewed journal and presented at conferences. Central results will be communicated to SVLFG and can be used to better understand insured persons, disseminate the information in health care campaigns and further improve health care services for farmers, foresters and gardeners. Access to the final trial dataset can be provided to fellow researchers upon request, depending on to be specified data security and data exchange regulation agreements.


  1. 1.

    Terhorst, Y., Braun, L., Titzler, I., Buntrock, C., Freund, J., Thielecke, J., Ebert, D. D., Baumeister H. Clinical and cost-effectiveness of a guided internet-based Acceptance and Commitment Therapy to improve chronic pain related disability in green professions (PACT-A): study protocol of a 36-month follow up pragmatic randomized controlled trial (under review).

  2. 2.

    Bührmann L, Schuurmans J, Ruwaard J, Genugten C van, Finch T, Rapley T, et al. Tailoring Implementation of Internet-based Cognitive Behavioural Therapy to Context-specific Needs across Europe and Australia. A Study Protocol for a Stepped-wedge Cluster Randomized Trial: the ImpleMentAll Project (under review).

  3. 3.

    Bührmann L, Schuurmans J, Ruwaard J, Genugten C van, Finch T, Rapley T, et al. Tailoring Implementation of Internet-based Cognitive Behavioural Therapy to Context-specific Needs across Europe and Australia. A Study Protocol for a Stepped-wedge Cluster Randomized Trial: the ImpleMentAll Project (under review).

  4. 4.

    Bührmann L, Schuurmans J, Ruwaard J, Genugten C van, Finch T, Rapley T, et al. Tailoring Implementation of Internet-based Cognitive Behavioural Therapy to Context-specific Needs across Europe and Australia. A Study Protocol for a Stepped-wedge Cluster Randomized Trial: the ImpleMentAll Project (under review).

  5. 5.

    Terhorst, Y., Braun, L., Titzler, I., Buntrock, C., Freund, J., Thielecke, J., Ebert, D. D., Baumeister H. Clinical and cost-effectiveness of a guided internet-based Acceptance and Commitment Therapy to improve chronic pain related disability in green professions (PACT-A): study protocol of a 36-month follow up pragmatic randomized controlled trial (under review).


  1. 1.

    Page H, Ayuso-Mateos J, Dowrick C, Vazquez-Barquero J. Depressive disorders in Europe: morbidity figures from the ODIN study. Br J Psychiatry. 2001;179(4):308–16.

    PubMed  Google Scholar 

  2. 2.

    Roy P, Tremblay G, Oliffe JL, Jbilou J, Robertson S. Male farmers with mental health disorders: a scoping review. Aust J Rural Health. 2013;21(1):3–7.

    PubMed  Google Scholar 

  3. 3.

    Sanne B, Mykletun A, Moen BE, Dahl AA, Tell GS. Farmers are at risk for anxiety and depression: the Hordaland health study. Occup Med (Chic Ill). 2004;54(2):92–100.

    CAS  Google Scholar 

  4. 4.

    Torske MO, Bjørngaard JH, Hilt B, Glasscock D, Krokstad S. Farmers’ mental health: a longitudinal sibling comparison - the HUNT study, Norway. Scand J Work Environ Health. 2016;42(6):547–56.

    PubMed  Google Scholar 

  5. 5.

    Judd F, Jackson H, Fraser C, Murray G, Robins G, Komiti A. Understanding suicide in Australian farmers. Soc Psychiatry Psychiatr Epidemiol. 2006;41(1):1–10.

    PubMed  Google Scholar 

  6. 6.

    Sanne B, Mykletun A, Dahl AA, Moen BE, Tell GS. Occupational differences in levels of anxiety and depression: the Hordaland health study. J Occup Environ Med. 2003;45(6):628–38.

    PubMed  Google Scholar 

  7. 7.

    Logstein B. Predictors of mental complaints among norwegian male farmers. Occup Med (Chic Ill). 2016;66(4):332–7.

    CAS  Google Scholar 

  8. 8.

    Onwuameze OE, Paradiso S, Peek-Asa C, Donham KJ, Rautiainen RH. Modifiable risk factors for depressed mood among farmers. Ann Clin Psychiatry. 2013;25(2):83–90.

    PubMed  Google Scholar 

  9. 9.

    Sue S, Keith H, Joan F, Aslög M. Stress in farmers: a survey of farmers in England and Wales. Occup Environ Med. 1998;55(11):729–34.

    Google Scholar 

  10. 10.

    Simkin S, Hawton K, Fagg J, Malmberg A. Stress in farmers: a survey of farmers in England and Wales. Occup Environ Med. 1998;55(11):729–34.

    CAS  PubMed  PubMed Central  Google Scholar 

  11. 11.

    Sachverständigenrat zur Begutachtung der Entwicklung im Gesundheitswesen (SVR). Gutachten 2014. Bedarfsgerechte Versorgung - Perspektiven für ländliche Regionen und ausgewählte Leistungsbereiche. (2014). Accessed 10 Oct 2018.

  12. 12.

    Bundespsychotherapeutenkammer. Ein Jahr nach der Reform der Psychotherapie-Richtlinie. Wartezeiten 2018. (2018). Accessed 1 Oct 2018.

    Google Scholar 

  13. 13.

    Andrews G, Issakidis C, Sanderson K, Corry J, Lapsley H. Utilising survey data to inform public policy: comparison of the cost-effectiveness of treatment of ten mental disorders record status study population. Br J Psychiatry. 2019;184(6):526–33.

    Google Scholar 

  14. 14.

    Horowitz JL, Garber J. The prevention of depressive symptoms in children and adolescents: a meta-analytic review. J Consult Clin Psychol. 2006;74(6):401.

    PubMed  Google Scholar 

  15. 15.

    Stockings EA, Degenhardt L, Dobbins T, Lee YY, Erskine HE, Whiteford HA, et al. Preventing depression and anxiety in young people: a review of the joint efficacy of universal, selective and indicated prevention. Psychol Med. 2016;46(1):11–26.

    CAS  PubMed  Google Scholar 

  16. 16.

    van Zoonen K, Buntrock C, Ebert DD, Smit F, Reynolds CF, Beekman AT, et al. Preventing the onset of major depressive disorder: a meta-analytic review of psychological interventions. Int J Epidemiol. 2014;43(2):318–29.

    PubMed  PubMed Central  Google Scholar 

  17. 17.

    Imamura K, Kawakami N, Furukawa TA, Matsuyama Y, Shimazu A, Umanodan R, et al. Effects of an Internet-Based Cognitive Behavioral Therapy (iCBT) Program in Manga Format on Improving Subthreshold Depressive Symptoms among Healthy Workers: A Randomized Controlled Trial. PLoS One. 2014;9(5):1–13.

  18. 18.

    Spek V, Nyklicek I, Smits N, Cuijpers P, Riper H, Keyzer J, et al. Internet-based cognitive behavioural therapy for subthreshold depression in people over 50 years old: a randomized controlled clinical trial. Psychol Med. 2007;37(12):1797–806.

    PubMed  Google Scholar 

  19. 19.

    Buntrock C, Ebert D, Lehr D, Riper H, Smit F, Cuijpers P, et al. Effectiveness of a web-based cognitive behavioural intervention for subthreshold depression: pragmatic randomised controlled trial. Psychother Psychosom. 2015;84(6):384–58.

    Google Scholar 

  20. 20.

    Buntrock C, Ebert DD, Lehr D, Smit F, Riper H, Berking M, et al. Effect of a Web-Based Guided Self-help Intervention for Prevention of Major Depression in Adults With Subthreshold Depression: A Randomized Clinical Trial. JAMA. 2016;315(17):1854–63.

  21. 21.

    Sander L, Rausch L, Baumeister H. Effectiveness of Internet-Based Interventions for the Prevention of Mental Disorders: A Systematic Review and Meta-Analysis. JMIR Ment Heal. 2016;3(3):1–20.

  22. 22.

    Kennedy AJ, Versace VL, Brumby SA. Research protocol for a digital intervention to reduce stigma among males with a personal experience of suicide in the Australian farming community. BMC Public Health. 2016;16(1):1–8.

  23. 23.

    Bundesministerium für Verkehr und digitale Infrastruktur. Aktuelle Breitbandverfügbarkeit in Deutschland (Stand Mitte 2018). (2018) Accessed July 7, 2019.

    Google Scholar 

  24. 24.

    Coughtrey AE, Pistrang N. The effectiveness of telephone-delivered psychological therapies for depression and anxiety: a systematic review. J Telemed Telecare. 2018;24(2):65–74.

    PubMed  Google Scholar 

  25. 25.

    Grol R, Grimshaw J. From best evidence to best practice: effective implementation of change in patients’ care. Lancet. 2003;362(9391):1225–30.

    PubMed  Google Scholar 

  26. 26.

    Grol R, Wensing M, Eccles M, Davis D. Improving patient care; the implementation of change in health care. 2nd ed. Oxford: Wiley Blackwell; 2013.

    Google Scholar 

  27. 27.

    Westfall JM, Mold J, Fagnan L. Practice-based research—“blue highways” on the NIH roadmap. JAMA. 2007;297(4):403–6.

    CAS  PubMed  Google Scholar 

  28. 28.

    Eisen JC, Marko-Holguin M, Fogel J, Cardenas A, Bahn M, Bradford N, et al. Pilot study of implementation of an internet-based depression prevention intervention (CATCH-IT) for adolescents in 12 US primary care practices: clinical and management/organizational behavioral perspectives. Prim Care Companion CNS Disord. 2013;15(6):1–27.

  29. 29.

    Minarik C, Moessner M, Ozer F, Bauer S. Implementierung und Dissemination eines internetbasierten Programms zur Prävention und frühen Intervention bei Essstörungen. Psychiatr Prax. 2013;40(06):332–8.

    PubMed  Google Scholar 

  30. 30.

    Auschra C, Deisner J, Berghöfer A, Sydow J. Sicherstellung der Gesundheitsversorgung in ländlich geprägten Regionen: Neue Organisationsmodelle und Maßnahmen. Berlin: Projektbericht in Herausgeberschaft der Stiftung Münch; 2018.

    Google Scholar 

  31. 31.

    Farrell SP, McKinnon CR. Technology and rural mental health. Arch Psychiatr Nurs. 2003;17(1):20–6.

    PubMed  Google Scholar 

  32. 32.

    Vallury KD, Jones M, Oosterbroek C. Computerized Cognitive Behavior Therapy for Anxiety and Depression in Rural Areas: A Systematic Review. J Med Internet Res. 2015;17(6):1–13.

  33. 33.

    Reddy P, Hernan AL, Vanderwood KK, Arave D, Niebylski ML, Harwell TS, et al. Implementation of diabetes prevention programs in rural areas: Montana and South-Eastern Australia compared. Aust J Rural Health. 2011;19(3):125–34.

    PubMed  Google Scholar 

  34. 34.

    Malmberg A, Simkin S, Hawton K. Suicide in Farmers. Br J Psychiatry. 1999;175(2):103–5.

    CAS  PubMed  Google Scholar 

  35. 35.

    Bradford NK, Caffery LJ, Smith AC. Awareness, experiences and perceptions of telehealth in a rural Queensland community. BMC Health Serv Res. 2015;15(1):427.

    PubMed  PubMed Central  Google Scholar 

  36. 36.

    Johansson AM, Lindberg I, Söderberg S. The views of health-care personnel about video consultation prior to implementation in primary health care in rural areas. Prim Health Care Res Dev. 2014;15(2):170–9.

    PubMed  Google Scholar 

  37. 37.

    Kirchner JE, Cody M, Thrush CR, Sullivan G, Rapp CG. Identifying factors critical to implementation of integrated mental health services in rural VA community-based outpatient clinics. J Behav Health Serv Res. 2004;31(1):13–25.

    PubMed  Google Scholar 

  38. 38.

    Jennett P, Jackson A, Ho K, Healy T, Kazanjian A, Woollard R, et al. The essence of telehealth readiness in rural communities: an organizational perspective. Telemed J E Health. 2005;11(2):137–45.

    PubMed  Google Scholar 

  39. 39.

    Keller PA, Murray JD, Hargrove DS. A rural mental Health Research agenda: defining context and setting priorities. J Rural Health. 1999;15(3):316–25.

    CAS  PubMed  Google Scholar 

  40. 40.

    Braun L, Titzler I, Ebert DD, Buntrock C, Terhorst Y, Freund J, et al. Clinical and cost-effectiveness of guided internet-based interventions in the indicated prevention of depression in green professions (PROD-A): study protocol of a 36-month follow-up pragmatic randomized controlled trial. BMC Psychiatry. 2019;19(1):1–16.

  41. 41.

    Thielecke J, Buntrock C, Titzler I, Braun L, Freund J, Berking M, et al. Clinical and cost-effectiveness of personalized tele-based coaching for farmers, foresters and gardeners to prevent depression: study protocol of an 18-month follow-up pragmatic randomized controlled trial (TEC-A). Front Psychiatry. 2020;11:125.

    PubMed  PubMed Central  Google Scholar 

  42. 42.

    Glasgow RE, Vogt TM, Boles SM. Evaluating the public health impact of health promotion interventions: the RE-AIM framework. Am J Public Health. 1999;89(9):1322–7.

    CAS  PubMed  PubMed Central  Google Scholar 

  43. 43.

    Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci. 2009;4(1):50.

    PubMed  PubMed Central  Google Scholar 

  44. 44.

    May CR, Mair F, Finch T, MacFarlane A, Dowrick C, Treweek S, et al. Development of a theory of implementation and integration: normalization process theory. Implement Sci. 2009;4(1):29.

    PubMed  PubMed Central  Google Scholar 

  45. 45.

    Cane J, O’Connor D, Michie S. Validation of the theoretical domains framework for use in behaviour change and implementation research. Implement Sci. 2012;7(1):37.

    PubMed  PubMed Central  Google Scholar 

  46. 46.

    Chaffin M. Commentary: organizational culture and practice epistemologies. Clin Psychol Sci Pract. 2006;13(1):90–3.

    Google Scholar 

  47. 47.

    Jung DI, Chow C, Wu A. The role of transformational leadership in enhancing organizational innovation: hypotheses and some preliminary findings. Leadersh Q. 2003;14(4):525–44.

    Google Scholar 

  48. 48.

    Weiner BJ, Amick H, Lee SD, Lee SD. Review measurement of organizational services research and other fields. Med Care Res Rev. 2008;65(4):379–436.

    PubMed  Google Scholar 

  49. 49.

    Pinnock H, Barwick M, Carpenter CR, Eldridge S, Grandes G, Griffiths CJ, et al. Standards for Reporting Implementation Studies (StaRI) Statement. BMJ. 2017;356(6795):1–9.

  50. 50.

    Aarons GA, Hurlburt M, Horwitz SM. Advancing a conceptual model of evidence-based practice implementation in public service sectors. Admin Pol Ment Health. 2011;38(1):4–23.

    Google Scholar 

  51. 51.

    Hoffmann TC, Glasziou PP, Boutron I, Milne R, Perera R, Moher D, et al. Better reporting of interventions: template for intervention description and replication (TIDieR) checklist and guide. BMJ. 2014;348:g1687.

    PubMed  Google Scholar 

  52. 52.

    Ebert DD, Buntrock C, Lehr D, Smit F, Riper H, Baumeister H, et al. Effectiveness of web- and Mobile-based treatment of subthreshold depression with adherence-focused guidance: a single-blind randomized controlled trial. Behav Ther. 2018;49(1):71–83.

    PubMed  Google Scholar 

  53. 53.

    Ebert DD, Heber E, Berking M, Riper H, Cuijpers P, Funk B, et al. Self-guided internet-based and mobile-based stress management for employees: results of a randomised controlled trial. Occup Environ Med. 2016;73(5):315–23.

    PubMed  Google Scholar 

  54. 54.

    Heber E, Lehr D, Ebert DD, Berking M, Riper H. Web-Based and Mobile Stress Management Intervention for Employees: A Randomized Controlled Trial. J Med Internet Res. 2016;18(1):1–15.

  55. 55.

    Buntrock C, Berking M, Smit F, Lehr D, Nobis S, Riper H, et al. Preventing depression in adults with subthreshold depression: Health-economic evaluation alongside a pragmatic randomized controlled trial of a web-based intervention. J Med Internet Res. 2017;19(1):1–16.

  56. 56.

    Nobis S, Ebert DD, Lehr D, Smit F, Buntrock C, Berking M, et al. Web-based intervention for depressive symptoms in adults with types 1 and 2 diabetes mellitus: A health economic evaluation. Br J Psychiatry. 2018;212(4):199–206.

  57. 57.

    Ebert DD, Lehr D, Heber E, Riper H, Cuijpers P, Scandinavian S, et al. Internet- and mobile-based stress managment for employees with adherence-focused guidance: efficacy and mechanism of change. Scand J Work Environ Health. 2016;42(5):382–94.

  58. 58.

    Boss L, Lehr D, Reis D, Vis C, Riper H, Berking M, et al. Reliability and Validity of Assessing User Satisfaction With Web-Based Health Interventions. J Med Internet Res. 2016;18(8):1–13.

  59. 59.

    Ebert DD, Berking M, Thiart H, Riper H, Laferton JAC, Cuijpers P, et al. Restoring depleted resources: efficacy and mechanisms of change of an internet-based unguided recovery training for better sleep and psychological detachment from work. Health Psychol. 2015;34S:1240–51.

    PubMed  Google Scholar 

  60. 60.

    Ebert DD, Nobis S, Lehr D, Baumeister H, Riper H, Auerbach RP, et al. The 6-month effectiveness of internet-based guided self-help for depression in adults with type 1 and 2 diabetes mellitus. Diabet Med. 2017;34(1):99–107.

    CAS  PubMed  Google Scholar 

  61. 61.

    Nobis S, Lehr D, Ebert DD, Baumeister H, Snoek F, Riper H, et al. Efficacy of a web-based intervention with mobile phone support in treating depressive symptoms in adults with type 1 and type 2 diabetes: a randomized controlled trial. Diabetes Care. 2015 May;38(5):776–83.

    PubMed  Google Scholar 

  62. 62.

    Thiart H, Lehr D, Ebert DD, Berking M, Riper H. Log in and breathe out: internet-based recovery training for sleepless employees with work-related strain - results of a randomized controlled trial. Scand J Work Environ Health. 2015 Mar;41(2):164–74.

    PubMed  Google Scholar 

  63. 63.

    Thiart H, Ebert DD, Lehr D, Nobis S, Buntrock C, Berking M, et al. Internet-Based Cognitive Behavioral Therapy for Insomnia: A Health Economic Evaluation. Sleep. 2016;39(10):1769–78.

  64. 64.

    Ebert DD, Kählke F, Buntrock C, Berking M, Smit F, Heber E, et al. A health economic outcome evaluation of an internet-based mobile-supported stress management intervention for employees. Scand J Work Environ Health. 2018;44(2):171–82.

  65. 65.

    Boß L, Lehr D, Schaub MP, Paz Castro R, Riper H, Berking M, et al. Efficacy of a web-based intervention with and without guidance for employees with risky drinking: results of a three-arm randomized controlled trial. Addiction. 2018;113(4):635–46.

    PubMed  Google Scholar 

  66. 66.

    Cuijpers P, Smit F. Subthreshold depression as a risk indicator for major depressive disorder: a systematic review of prospective studies. Acta Psychiatr Scand. 2004;109(5):325–31.

    CAS  PubMed  Google Scholar 

  67. 67.

    Riemann D, Voderholzer U. Primary insomnia: a risk factor to develop depression? J Affect Disord. 2003;76(1):255–9.

    PubMed  Google Scholar 

  68. 68.

    Plieger T, Melchers M, Montag C, Meermann R, Reuter M. Life stress as potential risk factor for depression and burnout. Burn Res. 2015;2(1):19–24.

    Google Scholar 

  69. 69.

    Parker G, Wilhelm K, Mitchell P, Austin M-P, Roussos J, Gladstone G. The influence of anxiety as a risk to early onset major depression. J Affect Disord. 1999;52(1):11–7.

    CAS  PubMed  Google Scholar 

  70. 70.

    Magni G, Moreschi C, Rigatti-Luchini S, Merskey H. Prospective study on the relationship between depressive symptoms and chronic musculoskeletal pain. Pain. 1994;56(3):289–97.

    CAS  PubMed  Google Scholar 

  71. 71.

    Saunders PA, Copeland JRM, Dewey ME, Davidson IA, Mcwilliam C, Sharma V, et al. Heavy drinking as a risk factor for depression and dementia in elderly men: findings from the Liverpool longitudinal community study. Br J Psychiatry. 1991;159(2):213–6.

    CAS  PubMed  Google Scholar 

  72. 72.

    Wang JL, Patten SB. Prospective study of frequent heavy alcohol use and the risk of major depression in the Canadian general population. Depress Anxiety. 2002;15(1):42–5.

    PubMed  Google Scholar 

  73. 73.

    Rotella F, Mannucci E. Diabetes mellitus as a risk factor for depression. A meta-analysis of longitudinal studies. Diabetes Res Clin Pract. 2013;99(2):98–104.

    CAS  PubMed  Google Scholar 

  74. 74.

    Mohr DC, Cuijpers P, Lehman K. Supportive accountability: A model for providing human support to enhance adherence to eHealth interventions. J Med Internet Res. 2011;13(1):1–11.

  75. 75.

    Lin J, Sander L, Paganini S, Schlicker S, Ebert D, Berking M, et al. Effectiveness and cost-effectiveness of a guided internet- and mobile-based depression intervention for individuals with chronic back pain: Protocol of a multi-centre randomised controlled trial. BMJ Open. 2017;7(12):1–13.

  76. 76.

    Nobis S, Lehr D, Ebert DD, Berking M, Heber E, Baumeister H, et al. Efficacy and cost-effectiveness of a web-based intervention with mobile phone support to treat depressive symptoms in adults with diabetes mellitus type 1 and type 2: design of a randomised controlled trial. BMC Psychiatry. 2013;13:1–10.

    Google Scholar 

  77. 77.

    Heber E, Ebert DD, Lehr D, Nobis S, Berking M, Riper H. Efficacy and cost-effectiveness of a web-based and mobile stress-management intervention for employees: design of a randomized controlled trial. BMC Public Health. 2013;13(1):1–12.

    Google Scholar 

  78. 78.

    Ebert DD, Lehr D, Baumeister H, Boß L, Riper H, Cuijpers P, et al. GET.ON Mood Enhancer: Efficacy of Internet-based guided self-help compared to psychoeducation for depression: An investigator-blinded randomised controlled trial. Trials. 2014;15(1):1–11.

  79. 79.

    Ebenfeld L, Kleine Stegemann S, Lehr D, Ebert DD, Jazaieri H, van Ballegooijen W, et al. Efficacy of a hybrid online training for panic symptoms and agoraphobia: study protocol for a randomized controlled trial. Trials. 2014;15(1):1–10.

    Google Scholar 

  80. 80.

    Thiart H, Lehr D, Ebert DD, Sieland B, Berking M, Riper H. Log in and breathe out: efficacy and cost-effectiveness of an online sleep training for teachers affected by work-related strain - study protocol for a randomized controlled trial. Trials. 2013;14(1):1–10.

    Google Scholar 

  81. 81.

    Heller HM, Hoogendoorn AW, Honig A, Broekman BFP, van Straten A. The effectiveness of a guided internet-based tool for the treatment of depression and anxiety in pregnancy (MamaKits online): randomized controlled trial. J Med Internet Res. 2020;22(3):e15172.

    PubMed  PubMed Central  Google Scholar 

  82. 82.

    Boeschoten RE, Nieuwenhuis MM, van Oppen P, Uitdehaag BMJ, Polman CH, Collette EH, et al. Feasibility and outcome of a web-based self-help intervention for depressive symptoms in patients with multiple sclerosis: a pilot study. J Neurol Sci. 2012;315(1–2):104–9.

    PubMed  Google Scholar 

  83. 83.

    Blake M, Waloszek JM, Schwartz O, Raniti M, Simmons JG, Blake L, et al. The SENSE study: post intervention effects of a randomized controlled trial of a cognitive–behavioral and mindfulness-based group sleep improvement intervention among at-risk adolescents. J Consult Clin Psychol. 2016;84(12):1039.

    PubMed  Google Scholar 

  84. 84.

    Ehrensaft MK, Knous-Westfall HM, Alonso TL. Web-based prevention of parenting difficulties in young, urban mothers enrolled in post-secondary education. J Prim Prev. 2016;37(6):527–42.

    PubMed  Google Scholar 

  85. 85.

    Willemse GRWM, Smit F, Cuijpers P, Tiemens BG. Minimal-contact psychotherapy for sub-threshold depression in primary care: randomised trial. Br J Psychiatry. 2004;185(5):416–21.

    PubMed  Google Scholar 

  86. 86.

    Bassler M, Potratz B, Krauthauser H. Der “helping Alliance questionnaire” (HAQ) von Luborsky. Psychotherapeut. 1995;40(1):23–32.

    Google Scholar 

  87. 87.

    Löwe B, Kroenke K, Herzog W, Gräfe K. Measuring depression outcome with a brief self-report instrument: sensitivity to change of the patient health questionnaire (PHQ-9). J Affect Disord. 2004;81(1):61–6.

    PubMed  Google Scholar 

  88. 88.

    Erbe D, Eichert HC, Rietz C, Ebert DD. Interformat reliability of the patient health questionnaire: validation of the computerized version of the PHQ-9. Internet Interv. 2016;5:1–4.

    PubMed  PubMed Central  Google Scholar 

  89. 89.

    Spitzer RL, Kroenke K, Williams JBW, Löwe B. A brief measure for assessing generalized anxiety disorder. Arch Intern Med. 2006;166(10):1092.

    PubMed  Google Scholar 

  90. 90.

    Lowe B, Decker O, Muller S, Brahler E, Schellberg D, Herzog W, et al. Validation and standardization of the generalized anxiety disorder screener (GAD-7) in the general population. Med Care. 2008;46(3):266–74.

    PubMed  Google Scholar 

  91. 91.

    Reis D, Lehr D, Heber E, Ebert DD. The German version of the perceived stress scale (PSS-10): evaluation of dimensionality, validity, and measurement invariance with exploratory and confirmatory Bifactor modeling. Assessment. 2017;26(7):1246–59.

    PubMed  Google Scholar 

  92. 92.

    Gierk B, Kohlmann S, Kroenke K, Spangenberg L, Zenger M, Braḧler E, et al. The somatic symptom Scale-8 (SSS-8): a brief measure of somatic symptom burden. JAMA Intern Med. 2014;174(3):399–407.

    PubMed  Google Scholar 

  93. 93.

    Kroenke K, Spitzer RL, Williams JBW. The PHQ-15: validity of a new measure for evaluating the severity of somatic symptoms. Psychosom Med. 2002;64(2):258–66.

    PubMed  Google Scholar 

  94. 94.

    Hawthorne G, Richardson J, Osborne R. The assessment of quality of life (AQoL) instrument: a psychometric measure of health-related quality of life. Qual Life Res. 1999;8(3):209–24.

    CAS  PubMed  Google Scholar 

  95. 95.

    Mittag O, Raspe H. Eine kurze Skala zur Messung der subjektiven Prognose der Erwerbstätigkeit: Ergebnisse einer Untersuchung an 4279 Mitgliedern der gesetzlichen Arbeiterrentenversicherung zu Reliabilität (Guttman-skalierung) und Validität der Skala. Rehabilitation. 2003;42(3):169–74.

    CAS  PubMed  Google Scholar 

  96. 96.

    Ladwig I, Rief W, Nestoriuc Y. Welche Risiken und Nebenwirkungen hat Psychotherapie? - Entwicklung des Inventars zur Erfassung Negativer Effekte von Psychotherapie (INEP). Verhaltenstherap. 2014;24(4):252–63.

    Google Scholar 

  97. 97.

    Devilly GJ, Borkovec TD. Psychometric properties of the credibility/expectancy questionnaire. J Behav Ther Exp Psychiatry. 2000;31(2):73–86.

    CAS  PubMed  Google Scholar 

  98. 98.

    Schmidt J, Wittmann WW. Fragebogen zur Messung der Patientenzufriedenheit. In: Brähler E, Schumacher J, Strauß B, editors. Diagnostische Verfahren in der Psychotherapie. Göttingen: Hogrefe Publishing GmbH; 2002. p. 392–6.

    Google Scholar 

  99. 99.

    Venkatesh V, Morris MG, Davis GB, Davis FD. User acceptance of information technology: toward a unified view. MIS Q Manag Inf Syst. 2003;27(3):425–78.

    Google Scholar 

  100. 100.

    Blum K. Patientenzufriedenheit bei ambulanten Operationen. Weinheim: Juventa; 1998.

    Google Scholar 

  101. 101.

    Shea CM, Jacobs SR, Esserman DA, Bruce K, Weiner BJ. Organizational readiness for implementing change: a psychometric assessment of a new measure. Implement Sci. 2014;9(1):7.

    PubMed  PubMed Central  Google Scholar 

  102. 102.

    Aarons GA, Ehrhart MG, Farahnak LR. The implementation leadership scale (ILS): development of a brief measure of unit level implementation leadership. Implement Sci. 2014;9(1):45.

    PubMed  PubMed Central  Google Scholar 

  103. 103.

    Aarons GA, Ehrhart MG, Torres EM, Finn NK, Roesch SC. Validation of the implementation leadership scale (ILS) in substance use disorder treatment organizations. J Subst Abus Treat. 2016;68:31–5.

    Google Scholar 

  104. 104.

    Finch, T.L., Girling, M., May, C.R., Mair, F.S., Murray, E., Treweek, S., Steen, I.N., McColl, E.M., Dickinson, C., Rapley T. NoMAD: Implementation measure based on Normalization Process Theory. [Measurement instrument] (German version ©2018 by the ImpleMentAll partners). 2015; Available from:

  105. 105.

    Elf M, Nordmark S, Lyhagen J, Lindberg I, Finch T, Åberg AC. The Swedish version of the normalization process theory measure S-NoMAD: translation, adaptation, and pilot testing. Implement Sci. 2018;13(1):146.

    PubMed  PubMed Central  Google Scholar 

  106. 106.

    Finch TL, Girling M, May CR, Mair FS, Murray E, Treweek S, et al. Improving the normalization of complex interventions: part 2-validation of the NoMAD instrument for assessing implementation work based on normalization process theory (NPT). BMC Med Res Methodol. 2018;18(1):135.

    PubMed  PubMed Central  Google Scholar 

  107. 107.

    Huijg JM, Gebhardt WA, Crone MR, Dusseldorp E, Presseau J. Discriminant content validity of a theoretical domains framework questionnaire for use in implementation research. Implement Sci. 2014;9:11.

    PubMed  PubMed Central  Google Scholar 

  108. 108.

    Palinkas LA, Aarons GA, Horwitz S, Chamberlain P, Hurlburt M, Landsverk J. Mixed method designs in implementation research. Adm Policy Ment Health Ment Health Serv Res. 2011;38(1):44–53.

    Google Scholar 

  109. 109.

    Meissner H, Creswell J, Klassen AC, Plano V, Smith KC. Best practices for mixed methods research in the health sciences. Methods. 2011;29:1–39.

    Google Scholar 

  110. 110.

    Tong A, Sainsbury P, Craig J. Consolidated criteria for reporting qualitative research (COREQ): a 32-item checklist for interviews and focus groups. Int J Qual Health Care. 2007;19(6):349–57.

    PubMed  Google Scholar 

  111. 111.

    R Core Team. R: A language and environment for statistical computing. Vienna; 2019. Available from:

  112. 112.

    Mayring P. Qualitative Inhaltsanalyse. In: Mey G, Mruck K, editors. Handbuch Qualitative Forschung in der Psychologie. Wiesbaden: VS Verlag für Sozialwissenschaften; 2010. p. 601–13.

    Google Scholar 

Download references


Authors would like to thank Friederike Dietz and Annika Montag for the creation of the Unipark surveys and their support in data security questions. Further, authors thank Sarah Banellis, Merle Bloom, Isabel Weber, Johanna Finitzer for their engagement in enrolling and supporting participants throughout the study. Authors thank the GET.ON, IVPNetworks and SVLFG staff who provide details on routine data, the consultation and enrolment process, as well as their interventions and Allison Grace for proofreading the manuscript.


The German insurance company SVLFG provided a financial contribution to the Friedrich-Alexander University Erlangen-Nürnberg and Ulm University as an expense allowance. SVLFG and the service provider had no role in study design, decision to publish or preparation of this manuscript. They will not be involved in data collection, analyses, decision to publish or preparation of future papers regarding the study. Open access funding provided by Projekt DEAL.

Author information




JF und IT share co-first authorship and contributed equally to this research and manuscript. DDE, HB and MB obtained funding for this study. IT, JF and DDE developed the design of the implementation study. IT, JF, JT and LB developed the design of the qualitative study with intervention participants. JF and IT drafted the manuscript and are responsible for recruitment, coordination of the trial and data collection. IT is supervising the trial management as operational lead of the project. All authors provided critical revision of the article and approved the final manuscript.

Corresponding authors

Correspondence to Johanna Freund or Ingrid Titzler.

Ethics declarations

Ethics approval and consent to participate

This study was approved by the Ethics Committee of the Friedrich-Alexander-University of Erlangen-Nürnberg (12.02.2019) and registered in the German Clinical Trials Register (DRKS00017078, HTML&TRIAL_ID = DRKS00017078) on April 18th, 2019.

Prior to their involvement, written informed consent for participation in the study will be obtained from all participants (insured persons, employees). SVLFG confirmed that informed consent document for the insured individuals will be stored securely at SVLFG. Informed consent documents of employees will be stored at the Friedrich-Alexander-University Erlangen-Nürnberg. All participant information will be stored securely in locked file cabinets and/or password-protected with restricted physical access in a secured server at the Friedrich-Alexander-University Erlangen-Nürnberg. All reports, data collection, and administrative forms will be identified exclusively by a study ID number assigned to each participant to maintain confidentiality. All records containing names or other personal identifiers, such as informed consent forms will be stored separately from study records identified by ID numbers. The list that links participant ID numbers to other identifying information will be stored in separate, password-protected files on external hardware with limited access.

Consent for publication

Not applicable.

Competing interests

IT reports to have received fees for lectures/workshops in the e-mental-health context from associations and training institutes for psychotherapists. IT is project lead of the trial site GET.ON for the European implementation research project ImpleMentAll funded by the European Commission. DDE has served as a consultant to/on the scientific advisory boards of Sanofi, Novartis, Minddistrict, Lantern, Schoen Kliniken, Ideamed and German health insurance companies (BARMER, Techniker Krankenkasse) and a number of federal chambers for psychotherapy. DDE and MB are stakeholders of the GET.ON Institute for health training online, which aims to implement scientific findings related to digital health interventions into routine care. HB has received consultancy fees and fees for lectures/workshops from chambers of psychotherapists and training institutes for psychotherapists in the e-mental-health context.

JF, JT, LB report no conflicts of interest.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Additional file 1.

Standards for Reporting Implementation Studies: the StaRI checklist for completion

Additional file 2.

Description of the GET.ON online health trainings

Additional file 3.

Description of the personalised tele-based coaching (IVPNetworks)

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit The Creative Commons Public Domain Dedication waiver ( applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Freund, J., Titzler, I., Thielecke, J. et al. Implementing internet- and tele-based interventions to prevent mental health disorders in farmers, foresters and gardeners (ImplementIT): study protocol for the multi-level evaluation of a nationwide project. BMC Psychiatry 20, 424 (2020).

Download citation


  • Implementation
  • Internet- and tele-based interventions
  • RE-AIM framework
  • Prevention
  • Mental health
  • Farmers