Skip to main content

The development and evaluation of a computerized decision aid for the treatment of psychotic disorders



Routinely monitoring of symptoms and medical needs can improve the diagnostics and treatment of medical problems, including psychiatric. However, several studies show that few clinicians use Routine Outcome Monitoring (ROM) in their daily work. We describe the development and first evaluation of a ROM based computerized clinical decision aid, Treatment-E-Assist (TREAT) for the treatment of psychotic disorders. The goal is to generate personalized treatment recommendations, based on international guidelines combined with outcomes of mental and physical health acquired through ROM. We present a pilot study aimed to assess the feasibility of this computerized clinical decision aid in daily clinical practice by evaluating clinicians’ experiences with the system.


Clinical decision algorithms were developed based on international schizophrenia treatment guidelines and the input of multidisciplinary expert panels from multiple psychiatric institutes. Yearly obtained diagnostic (ROM) information of patients was presented to treating clinicians combined with treatment suggestions generated by the algorithms of TREAT. In this pilot study 6 clinicians and 16 patients of Lentis Psychiatric Institute used the application. Clinicians were interviewed and asked to fill out self-report questionnaires evaluating their opinions about ROM and the effectiveness of TREAT.


Six clinicians and 16 patients with psychotic disorders participated in the pilot study. The clinicians were psychiatrists, physicians and nurse-practitioners which all worked at least 8 years in mental health care of which at least 3 years treating patients with psychotic illnesses. All Clinicians found TREAT easy to use and would like to continue using the application. They reported that TREAT offered support in using diagnostic ROM information when drafting the treatment plans, by creating more awareness of current treatment options.


This article presents a pilot study on the implementation of a computerized clinical decision aid linking routine outcome monitoring to clinical guidelines in order to generate personalized treatment advice. TREAT was found to be feasible for daily clinical practice and effective based on this first evaluation by clinicians. However, adjustments have to be made to the system and algorithms of the application. The ultimate goal is to provide appropriate evidence based care for patients with severe mental illnesses.

Peer Review reports


Treatment of psychotic disorders

Almost 1 % of the population in the western world will eventually fulfil the criteria of schizophrenia or a related severe mental illness [1]. Core symptoms of many people suffering from psychotic disorders are hallucinations, delusions, incoherent thoughts, memory problems, loss of initiative, flat affect, poverty of speech and social withdrawal [2]. Moreover, patients frequently experience problems with psychosocial functioning, such as a lack of daytime activities, social contacts, intimate relationships and a reduced quality of life [3, 4]. They often have poor physical health and experience medication side effects that contribute to an early onset of cardiovascular diseases. Different studies have shown a reduced life expectancy ranging from 10 up to 28 years [5, 6]. Some patients manage to recover both in terms of their symptoms, as well as in reaching personal and social goals. However the majority only partially recovers, with recurrence of symptoms and enduring personal and social problems often for the rest of their life. Especially patients with the most severe symptoms (fulfilling criteria for schizophrenia or schizoaffective disorders) often need lifetime medical, psychiatric and social care. Recommended treatment options are described in national treatment guidelines; in the Netherlands the Multidisciplinary Guideline for Schizophrenia is used (which is largely in line with the NICE guideline) [7]. The Optimal Treatment Project revealed that 2 years of optimal, evidence-based treatment led towards a clear trend in recovery from clinical impairment and social disability of patients with psychotic disorders [8]. Despite increasing evidence that pharmacological and psychosocial interventions are effective in improving clinical symptoms and patients’ functioning, the availability of treatment interventions and integration in psychiatric care is often suboptimal [9]. Also, many patients with psychotic disorders find it difficult to express their needs, show a decreased awareness of their symptoms and only partially understand the different possible treatment options. Therefore, psychological, medical and social problems often go undetected or untreated [10]. There is a challenge to monitor symptoms and unmet care needs of these patients in order to offer optimal care, especially in realizing their varying needs in different domains for many years.

Routine outcome monitoring

Routine outcome monitoring (ROM) is one such way to monitor symptoms and care needs. ROM can be described as the use of standardized instruments to systematically and repeatedly measure different aspects of patients’ symptoms, health, social functioning and wellbeing in order to improve their treatment [11, 12]. For patients with schizophrenia and related mental health problems, regular participation in ROM contributes to systematic evaluation of their varying needs in (mental) health care over many years. This could offer these patients relevant treatment options adjusted to actual needs. However, only few clinicians use ROM data in their day-to-day work [13, 14]. In the Northern Netherlands, an extensive ROM screening called the Pharmacotherapy Monitoring and Outcome Survey (ROM-Phamous), consisting of a large array of instruments [15], has been implemented since 2007. Its main target is to identify needs of care in psychiatric, medical and social domains in order to optimize the treatment of patients with psychotic disorders. The data obtained also allows for scientific research. Although it has been shown that ROM-Phamous successfully helps to identify unmet needs, it is still not optimally used in clinical decision making and in offering recommended evidence-based treatment options to people with psychotic disorders [16,17,18].

Treatment E-assist

There is often not just a single best option when making treatment decisions in healthcare. Different treatment options may have varying risks and benefits, making it challenging to offer the optimal option when decisions are sensitive to personal preference. Clinical decision aids (CDAs) are evidence-based tools to support decision making in healthcare and have been gaining popularity in various medical disciplines [19]. A recent meta-analysis shows CDAs improve patients’ knowledge about available treatment options, facilitate accurate risk perception and increase their active involvement in the decision making process [20]. While knowledge about effective mental health care keeps growing, translation to daily clinical practice is lagging [21, 22]. CDAs can serve as a guideline implementation strategy by transferring evidence-based knowledge to day-to-day patient care. Despite the potential benefits of CDAs, their use in mental healthcare is very limited. In one study, a computerized CDA linking patient specific data to guidelines led to a decrease of symptoms and lowered re-hospitalization rates among people with psychotic disorders [23]. However, all diagnostic measurements for this CDA had to be collected by clinicians themselves, making the process time consuming. In the current study we describe Treatment E-Assist (TREAT). This is a recently developed computerized CDA that combines diagnostic patient data, collected using ROM-Phamous, with guidelines. TREAT facilitates the use of ROM-Phamous in daily clinical practice by summarising patients’ unmet needs. As a second step, evidence-based treatment recommendations based on the Multidisciplinary Guideline for Schizophrenia are generated to assist the clinician and patient to make shared decisions about these unmet needs.

Research aim

This article describes the development of TREAT and the results of the pilot study, evaluating TREAT, as a computerized CDA designed for the treatment of patients with psychotic disorders. The pilot study tests the feasibility of TREAT in daily clinical practice by evaluating clinicians’ experiences when working with the application.


Development of TREAT

Substantive design

The algorithms of TREAT are based on ROM-Phamous data and the Multidisciplinary Guideline for Schizophrenia [7]. The algorithms were designed in collaboration with two multidisciplinary expert panels, both consisting of 7 members representing different institutions. These panels included researchers, psychiatrists, psychologists and nurse-practitioners, all experienced in working with ROM-Phamous. The first panel focused on somatic problems and pharmacotherapy and the second panel focused on psychosocial interventions. Participants from the expert panels did not participate in the pilot study described hereafter.

The first session of the expert panels was an introduction of the project and a brainstorm to collect first ideas and thoughts. Next, researchers proposed possible problematic domains in care for people with a psychotic disorder: positive symptoms, negative symptoms, cognitive symptoms, psychosocial problems, and somatic problems. Each domain was further divided into subcategories. The researchers selected matching items from the ROM-Phamous instruments for each subcategory and proposed cut-off scores. TREAT displays these subcategories as problematic when patient measures on the matching instruments exceed the cut-off scores. The draft proposal was then discussed with the panels, until consensus was reached. Cut-off scores were based on expert opinions when explicit guidelines were lacking. Instead of the Multidisciplinary Guideline for Schizophrenia [7], the pharmacotherapy panel decided to use a more detailed guideline for the treatment of cardiovascular risk factors based on the input of somatic doctors working in psychiatry (of Mental Health Care Center Drenthe, The Netherlands). This guideline (Guideline for Cardiovascular Risk Management Drenthe) offers cut-off scores and treatment recommendations on cardiovascular risk factors for patients who have been using antipsychotic medication for a long period of time. Finally, the researchers proposed treatment recommendations to both panels for each care domain, based on the Multidisciplinary Guideline for Schizophrenia [7]. Treatment recommendations were complemented where necessary and discussed until consensus was reached. The final TREAT algorithms were discussed with two guideline experts (HK and SC) to assess whether the Multidisciplinary Guideline for Schizophrenia [7], had been properly followed. Figure 1 depicts a schematic of TREAT.

Fig. 1
figure 1

Schematic of TREAT

Program design

ROM-Phamous and TREAT were both developed by a company (RoQua) that specializes in privacy protecting ROM systems that are accessed via the electronic patient record. TREAT has been built as an addition to the ROM-Phamous system and generates an interactive (Dutch) report for each individual patient. The first part of the report displays a summary of the ROM results. It contains a graph depicting symptom dimensions, treatment effects, patient satisfaction and unmet needs in different areas of life, such as mental health, physical health and antipsychotic medication. It also depicts general information about the patient, for instance the types of treatment the patient has received during the previous year. Finally, an overview is depicted of all care domains that are measured by ROM highlighting (in blue) which areas might (still) be problematic for a patient as depicted in Fig. 2. When a clinician selects a care domain, the relevant instruments, items, scores and treatment recommendations are displayed. In the graphs, the color of a bar indicates the severity of the problem (green = no problem, orange = potential problem and red = problem). On each page that depicts a care domain, one can click on the “back to overview” button to move back to the summary page. Clinicians can navigate through the TREAT report to assess all relevant ROM-Phamous results and treatment recommendations of a patient as depicted in Figs. 2 and 3.

Fig. 2
figure 2

Screenshot of TREAT 1

Fig. 3
figure 3

Screenshot of TREAT 2

Pilot study


Six clinicians and sixteen patients of two (outpatient) Functional Assertive Community Treatment (FACT) teams of Lentis Psychiatric Institute participated in the pilot study. The clinicians were three psychiatrists, two nurse-practitioners and one physician. Patients were eligible for the study when they had a DSM IV diagnosis on the psychosis spectrum, or a personality or mood disorder with psychotic features. All patients filled out an informed consent form. The procedures were in accordance with the declaration of Helsinki as confirmed by the Medical Ethics Committee of the University Medical Center Groningen.


Before the clinicians started using TREAT, they were asked to fill out a questionnaire that assessed their opinions about ROM-Phamous. Next, each clinician used TREAT before and/or during the discussion of the ROM results with a patient. The first three patients with psychotic disorders scheduled to have an appointment with their clinician to discuss their yearly ROM results, were asked to participate in the study. One of the six clinicians only participated with one patient due to planning issues during the time of this pilot study and patients declining to participate. The nurse who performed the ROM-Phamous screening was instructed to create the individual TREAT report. Both the nurses and the clinicians received instructions about TREAT from a researcher (MT). During these hour long treatment sessions, ROM results are discussed and treatment plans are drafted or adjusted. Afterwards, clinicians and patients filled out a questionnaire about the clinical decision making process during the sessions. This included questions about the topics that were discussed and the treatment options that were considered. At the end of the pilot study, clinicians filled out a questionnaire and participated in a brief open interview that both assessed their experiences with TREAT.


Assessments were made with two self-developed, theory-based Dutch questionnaires. The first one was the ‘ROM State-of-Mind’ consisting of 24 items, with the first 22 items assessing participants’ acceptance of ROM-Phamous on a Likert scale from 1 (completely disagree) to 5 (completely agree). Item 23 assessed overall appreciation of ROM-Phamous on a scale from 1 (very poorly) to 10 (excellent) and item 24 was an open question about suggestions and comments regarding ROM-Phamous. Items constituted seven subscales, of which five had a high internal consistency (Cronbach’s Alpha ≥ .7) and two a low internal consistency (Cronbach’s Alpha < .4) [18].

Secondly, participants filled out the ‘TREAT State-of-Mind’ consisting of 27 items assessing statements about TREAT, each rated on a Likert scale from 1 (completely disagree) to 5 (completely agree). The items constituted eight subscales, measuring usage behaviour, support, power, issue-impact, emotion, ease of use, usefulness, and facilitating conditions. The following two items assessed the acceptance of TREAT in general and the integration of both TREAT and ROM-Phamous. Both items were rated on a scale from 1 (very poorly) to 10 (excellent). The next item was an open question to collect ideas, comments or suggestions about TREAT.

The questionnaire also registered the amount of times clinicians consulted TREAT and for how long they had used TREAT on average per consult. The remaining six items assessed clinicians’ characteristics: profession, department, gender, age, and number of years working in mental health care and psychosis care.

We used the Clinical Decision Making in Routine Care (CDRC) questionnaire [24], to assess the treatment sessions. This questionnaire was translated in to Dutch and expanded with specific categories, largely representing the problems that are assessed with TREAT (see Additional file 1). The CDRC has a staff and patient version, consisting of 22 and 21 items each. The original authors, CEDAR study group, provided permission to use the questionnaire.


Clinicians’ experiences with ROM and TREAT

All clinicians had worked with ROM-Phamous for 7 years on average (ranging from 3 years to 9 years).

They were generally positive about ROM-Phamous, grading it with an average of 7 (SD 0.89) on a scale from 1 (poor) to 10 (excellent). They have positive feelings towards ROM-Phamous (subscale ‘emotion’), see it as a useful addition to their job (subscale ‘usefulness’), but don’t find it very easy to use (subscale ‘ease of use’). Results are depicted in Table 1. Clinicians were also positive about TREAT with an average rating of 7.5 (SD 0.84) and the integration of TREAT and ROM with an average rating of 7.3 (SD 1.37) on a scale from 1 (poor) to 10 (excellent). Most clinicians had used TREAT at least three times during the pilot study, with the exception of one clinician who only worked with TREAT a single time. The time TREAT was used differed per user: 5 to 15 min (1 psychiatrist), 15 to 30 min (1 nurse-practitioner and 1 physician) and more than 30 min per session (2 psychiatrist and 1 nurse-practitioner). Most clinicians find TREAT useful and expect it will help them improve their work. They state that it helps to interpret the (diagnostical) ROM-Phamous results and offers support in drafting the treatment plan. It also enhances awareness of the existing treatment options (subscale ‘usefulness’). TREAT fits with good clinical care, clinicians are proud of its development (subscale ‘emotion’) and expect to use TREAT in the future (subscale ‘usage behaviour’). TREAT is thought to be easy to use and requires little mental effort. Opinions about the lay-out of TREAT are mixed (subscale ‘ease of use’), for example some clinicians preferred the graphs displaying data vertically whilst others preferred it horizontally. Most clinicians state that they have enough time to use TREAT in daily clinical practice and that it helps them to work more efficiently (subscale ‘facilitating conditions’). They do not think TREAT will have impact on their professional autonomy (subscale ‘power’), on their job or on patient care in general (subscale ‘issue-impact’). Results are depicted in Table 2. Some clinicians preferred to use TREAT on the computer while others preferred to print the information. The topics that were mentioned in the open interview are depicted in Table 3.

Table 1 Scores on the ROM State-of-Mind questionnaire (clinicians)
Table 2 Scores on the TREAT State-of-Mind questionnaire (clinicians)
Table 3 Topics mentioned in the brief open interview about TREAT (clinicians)

Feasibility of questionnaires

Both clinicians and patients completed the CDRC questionnaires. All clinicians filled out the staff version of the CDRC but some of the patients experienced difficulties with their version of the questionnaire. They remembered most of the topics discussed in the feedback sessions but some had trouble to be specific or to categorize the different topics or mention specific treatment options that were suggested.


A ROM inventory in 2011, for the Dutch ministry of health, welfare and sport, recommended combining ROM to CDAs to improve the treatment process within mental healthcare [25]. Despite these recommendations such systems are currently unavailable. With TREAT, a computerized CDA has been developed linking ROM to treatment guidelines. In this way TREAT generates personalized treatment advice for the treatment of patients with psychotic disorders. The primary aim of this pilot study was to describe the development of TREAT and assess its feasibility for daily clinical practice.

Clinicians’ evaluations

In general clinicians were positive about working with TREAT. Most of them found the system easy to use without requiring extra mental effort or time. This is important as usability and limited time investment are the two most important factors affecting successful implementation of CDAs [26]. One clinician did prefer the printed TREAT report for the feedback sessions because this person felt that the computer could disturb communication with patients. However, concerns from a previous study that computer use would be distracting, decrease eye contact or depersonalize the interaction [27], were not replicated. All clinicians indicated they would like to continue working with TREAT as they found it fitting with good clinical practice. Some concerns were expressed about the recommendations being or becoming too familiar over time. Lack of novelty could potentially stop clinicians from working with the application. In contrast, some clinicians became more aware of guidelines and discovered new treatment options. There is growing evidence that treatment in accordance with guidelines within mental healthcare, can positively affect patient care [8, 28, 29]. One clinician suggested that the ROM-Phamous results of previous years should be incorporated in order to evaluate changes over time. This exemplifies that the TREAT algorithms can be improved and tailored even more to individual patients. CDAs sometimes fail to take contextual information into account or have algorithms that insufficiently fit complex patient scenarios [30]. When asking clinicians about their opinions regarding ROM-Phamous, they indicated that the outcomes are not always easy to use for clinical decision making. This is in line with previous studies showing that the outcomes of ROM-Phamous are not used to its full potential in daily clinical practice [17, 18, 31]. TREAT simplifies the interpretation of these outcomes and facilitates a basis for more explicit decision making. Future research should focus on the effects of TREAT in the clinical decision making process. Lessons learned can improve the TREAT application, but may also help to develop other CDA’s.

Future adjustments & research

Adequately informing patients about their health and available treatment options is the future of healthcare, in which CDAs should play a pivotal role. While CDAs are commonly used in medical fields such as oncology, orthopedics and cardiology, its use in mental healthcare is still very rare. With TREAT a high quality and easy to use CDA is now available in this field. Some adjustments will be made to the system based on the results of this pilot study. For instance the algorithms will be updated to ensure that TREAT reports can still be generated when part of the data is missing. ROM-Phamous consists of multiple instruments so patients are not always able to complete all questionnaires. Furthermore, two printable versions of the TREAT report will be added, namely a summary and the complete report. These adjustments will enable a large follow-up study aiming to investigate the effects of TREAT on clinical decision making. Because there are hardly any validated methodologies available to assess the content of treatment sessions in mental healthcare, modified and translated versions of the CDRC questionnaires were tested. Clinicians were able to fill out their version. However, patients experienced difficulties in categorizing the different topics that were discussed during the treatment sessions. These difficulties probably reflect cognitive difficulties in line with their psychiatric problems. Therefore, we will use only the staff version of the CDRC questionnaire for future research. Clinicians reported that patients appreciated the TREAT report and that it could improve shared decision making. Previous research in somatic medicine supports the notion that CDAs are effective in improving shared decision-making [20]. Recently there have been calls for widespread access to CDAs to improve the level of shared decision-making within mental healthcare [32]. Patients prefer an active role in the decision-making process and are more likely to adhere to their treatment plans if they actively participated in the draft [33]. The follow-up study is designed to assess whether CDA’s, in this case TREAT, can indeed increase shared decision making within a mental healthcare setting. So far research on computerized CDAs shows that they may improve disease management and diagnostics, however the effects on patients’ functioning and final clinical outcome remains unclear [34]. Assessing the effects of TREAT on patients’ symptoms, physical health and psychosocial functioning, will be another important goal of the follow-up study. If a CDA like TREAT is beneficial in treatment of psychotic disorders, it might be worthwhile to develop similar systems for different patient groups.

Strengths and limitations

To the best of our knowledge, this study presents the first computerized CDA combining ROM and treatment guidelines in an electronic patient record within mental health care. TREAT was developed in close collaboration with healthcare professionals. This is a strength of TREAT, as sufficient knowledge of developers about the target group of CDAs is important for successful implementation [26]. ROM is sometimes experienced as behavioural control by some users who feel obliged by political or financial motives [18]. This was not replicated in previous or the current study [17, 18], as all clinicians had positive opinions about ROM-Phamous. A limitation of the current study is the small sample of only six clinicians from the same psychiatric institute. Although large enough to test the applicability of TREAT in daily clinical practice, a larger sample of clinicians from multiple centres are preferred when testing the effects of the application on the clinical decision making process and patient outcomes.


This pilot study describes the development and first evaluation of a computerized CDA (TREAT) linking ROM to clinical guidelines to generate personalized treatment recommendations in the treatment of psychotic disorders. The primary aim was to assess the feasibility of TREAT for daily clinical practice by evaluating clinicians’ opinions when working with the system. In sum, clinicians found TREAT useful for daily clinical practice, easy to use, fitting with good clinical care and all of them would like to use the system in the future. TREAT was not felt to reduce clinicians’ professional autonomy nor was it perceived as behavioural control. Clinicians expect TREAT to have a positive impact on their daily job but are unsure if it will improve patient outcomes, such as reduced symptoms and improved psychosocial wellbeing. The application will be adjusted and improved for a follow-up study based on the results from this pilot. The follow-up study will investigate the effects of TREAT on shared decision-making, the clinical decision-making process and patient outcomes.



Clinical decision aids


Clinical Decision Making in Routine Care


Functional Assertive Community Treatment


Routine outcome monitoring


Routine outcome monitoring - pharmacotherapy monitoring and outcome survey




  1. McGrath J, Saha S, Chant D, Welham J. Schizophrenia: a concise overview of incidence, prevalence, and mortality. Epidemiol Rev. 2008;30:67–76.

    Article  PubMed  Google Scholar 

  2. Cahn W, Krabbendam L, Myin-Germeys I, Bruggeman R, De Haan L. Diagnostiek van het psychiatrische toestandsbeeld (Dutch): Handboek Schizofrenie (Dutch). Utrecht: De Tijdstroom Uitgeverij; 2011. p. 62–3.

  3. Thornicroft G, Tansella M, Becker T, Knapp M, Leese M, Schene A, Vazquez-Barquero JL. The personal impact of schizophrenia in Europe. Schizophr Res. 2004;69:125–32.

    Article  PubMed  Google Scholar 

  4. Fleischhacker W, Cetkovich-Bakmas M, De Hert M, Hennekens C, Lambert M, Leucht S, et al. Comorbid somatic illnesses in patients with severe mental disorders: clinical, policy, and research challenges. J Clin Psychiatry. 2008;69:514–9.

    Article  PubMed  Google Scholar 

  5. Laursen TM, Munk-Olsen T, Vestergaard M. Life expectancy and cardiovascular mortality in persons with schizophrenia. Curr Opin Psychiatry. 2012;25:83–8.

    Article  PubMed  Google Scholar 

  6. Olfson M, Gerhard T, Huang C, Crystal S, Stroup TS. Premature mortality among adults with schizophrenia in the United States. JAMA Psychiatry. 2015;72:1172–81.

    Article  PubMed  Google Scholar 

  7. Alphen C, Ammeraal M, Blanke C, Boonstra N, Boumans H, Bruggeman R, et al. Multidisciplinaire richtlijn schizofrenie. Utrecht: De Tijdstroom Uitgeverij; 2012.

  8. Falloon IR, Montero I, Sungur M, Mastroeni A, Malm U, Economou M, et al. Implementation of evidence-based treatment for schizophrenic disorders: two-year outcome of an international field trial of optimal treatment. World Psychiatry. 2004;3:104–9.

    PubMed  PubMed Central  Google Scholar 

  9. Chien WT, Leung SF, Yeung FK, Wong WK. Current approaches to treatments for schizophrenia spectrum disorders, part II: psychosocial interventions and patient-focused perspectives in psychiatric care. Neuropsychiatr Dis Treat. 2013;9:1463–81.

    Article  PubMed  PubMed Central  Google Scholar 

  10. Cradock-O'Leary J, Young AS, Yano EM, Wang M, Lee ML. Use of general medical services by VA patients with psychiatric disorders. Psychiatr Serv. 2002;53:874–8.

    Article  PubMed  Google Scholar 

  11. Trauer T. Outcome measurement in chronic mental illness. Int Rev Psychiatry. 2010;22:99–113.

    Article  PubMed  Google Scholar 

  12. Carlier IV, Meuldijk D, van Vliet IM, van Fenema EM, van der Wee NJ, Zitman FG. Empirical evidence for the effectiveness of routine outcome monitoring. A study of the literature. Tijdschr Psychiatr. 2012;54:121–8.

    PubMed  CAS  Google Scholar 

  13. Gilbody S, House A, Sheldon T. Psychiatrists in the UK do not use outcome measures. National survey. Br J Psychiatry. 2002;180:101–3.

    Article  PubMed  Google Scholar 

  14. Zimmerman M, McGlinchey JB. Why don’t psychiatrists use scales to measure outcome when treating depressed patients? J Clin Psychiatry. 2008;69:1916–9.

    Article  PubMed  Google Scholar 

  15. Bartels-Velthuis A, Visser E, Arends J, Pijnenborg G, Wunderink L, Jorg F, et al. Towards a comprehensive routine outcome monitoring program for people with psychotic disorders: the Pharmacotherapy Monitoring and Outcome Suvery (PHAMOUS) [Epub ahead of print].

  16. Boswell JF, Kraus DR, Miller SD, Lambert MJ. Implementing routine outcome monitoring in clinical practice: benefits, challenges, and solutions. Psychother Res. 2015;25:6–19.

    Article  PubMed  Google Scholar 

  17. Tasma M, Swart M, Wolters G, Liemburg E, Bruggeman R, Knegtering H, et al. Do routine outcome monitoring results translate to clinical practice? A cross-sectional study in patients with a psychotic disorder. BMC Psychiatry. 2016;16:107. 016–0817-6

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  18. Tasma M, Liemburg EJ, Knegtering H, Delespaul PAEG, Boonstra A, Castelein S. Exploring the use of routine outcome monitoring in the treatment of patients with a psychotic disorder. Eur Psychiatry. 2017;42:89–94.

    Article  PubMed  CAS  Google Scholar 

  19. O'Connor A, Llewellyn-Thomas H, Stacey D. International patient decision aid standards (IPDAS). Collaboration background document. 2005.

  20. Stacey D, Legare F, Lewis K, Barry MJ, Bennett CL, Eden KB, et al. Decision aids for people facing health treatment or screening decisions. Cochrane Database Syst Rev. 2017;4:CD001431.

    PubMed  Google Scholar 

  21. Grol R. Knowledge transfer in mental health care: how do we bring evidence into day-to-day practice? Can J Psychiatr. 2008;53:275–6.

    Article  Google Scholar 

  22. Bauer MS. A review of quantitative studies of adherence to mental health clinical practice guidelines. Harv Rev Psychiatry. 2002;10:138–53.

    Article  PubMed  Google Scholar 

  23. Janssen B, Ludwig S, Eustermann H, Menke R, Haerter M, Berger M, et al. Improving outpatient treatment in schizophrenia: effects of computerized guideline implementation--results of a multicenter-study within the German research network on schizophrenia. Eur Arch Psychiatry Clin Neurosci. 2010;260:51–7.

    Article  PubMed  CAS  Google Scholar 

  24. Puschner B, Neumann P, Jordan H, Slade M, Fiorillo A, Giacco D, et al. Development and psychometric properties of a five-language multiperspective instrument to assess clinical decision making style in the treatment of people with severe mental illness (CDMS). BMC Psychiatry. 2013;13:48. 244X-13-48

    Article  PubMed  PubMed Central  Google Scholar 

  25. Van der Feltz-Cornelis C, Andrea H, Kessels E, Duivenvoorden H, Biemans H, Metz M. Does routine outcome monitoring have a promising future? An investigation into the use of shared decision-making combined with ROM for patients with a combination of physical and psychiatric symptoms. Tijdschrift voor Psychiatrie. 2014;56:375–84.

  26. Kux BR, Majeed RW, Ahlbrandt J, Rohrig R. Factors influencing the implementation and distribution of clinical decision support systems (CDSS). Stud Health Technol Inform. 2017;243:127–31.

    PubMed  Google Scholar 

  27. Trivedi MH, Daly EJ, Kern JK, Grannemann BD, Sunderajan P, Claassen CA. Barriers to implementation of a computerized decision support system for depression: an observational report on lessons learned in “real world” clinical settings. BMC Med Inform Decis Mak. 2009;9:6. 6947–9-6

    Article  PubMed  PubMed Central  Google Scholar 

  28. Girlanda F, Fiedler I, Ay E, Barbui C, Koesters M. Guideline implementation strategies for specialist mental healthcare. Curr Opin Psychiatry. 2013;26:369–75.

    Article  PubMed  Google Scholar 

  29. Girlanda F, Fiedler I, Becker T, Barbui C, Koesters M. The evidence-practice gap in specialist mental healthcare: systematic review and meta-analysis of guideline implementation studies. Br J Psychiatry. 2017;210:24–30.

    Article  PubMed  Google Scholar 

  30. Miller A, Moon B, Anders S, Walden R, Brown S, Montella D. Integrating computerized clinical decision support systems into clinical work: a meta-synthesis of qualitative research. Int J Med Inform. 2015;84:1009–18.

    Article  PubMed  Google Scholar 

  31. Bruins J, Pijnenborg GH, van den Heuvel ER, Visser E, Corpeleijn E, Bartels-Velthuis AA, et al. Persistent low rates of treatment of metabolic risk factors in people with psychotic disorders: a PHAMOUS study. J Clin Psychiatry. 2017;

  32. Slade M. Implementing shared decision making in routine mental health care. World Psychiatry. 2017;16:146–53.

    Article  PubMed  PubMed Central  Google Scholar 

  33. De Las Cuevas C, Penate W, de Rivera L. To what extent is treatment adherence of psychiatric patients influenced by their participation in shared decision making? Patient Prefer Adherence. 2014;8:1547–53.

    Article  PubMed  PubMed Central  Google Scholar 

  34. Garg AX, Adhikari NK, McDonald H, Rosas-Arellano MP, Devereaux PJ, Beyene J, et al. Effects of computerized clinical decision support systems on practitioner performance and patient outcomes: a systematic review. JAMA. 2005;293:1223–38.

    Article  PubMed  CAS  Google Scholar 

Download references


We would like to thank the healthcare professionals who participated in the expert panels for the development of the algorithms of TREAT; Johan Arends (psychiatrist) of Mental Health Care Center Drenthe, Frank van Es (psychiatrist) of the University Center for Psychiatry of the University Medical Center Groningen, and the clinicians of Lentis Psychiatric Institute; Martin de Jonge (psychiatrist), Sigrid van der Linden (psychiatrist), Wouter Obelink (nurse practitioner), Gert Wolters (nurse practitioner), Marloes van der Kley (nurse), Gerard Lohuis (nurse practitioner), Abra Havinga (psychologist) and Wouter Draaisma (psychologist). We would also like to thank patients and clinicians of Lentis Psychiatric Institute for participating in our study and Anneke Zijlstra and Magda Poldervaart (the nurses who perform the ROM-screenings) for their help in coordinating the pilot study. Finally we would like to thank Jojanneke Bruins (researcher) for her critical review of this manuscript.


This work was financially supported by the (Dutch) ROOS foundation (Stichting ROOS).

Availability of data and materials

The authors approve on making available the datasets used in this study, when requested to the corresponding author upon reasonable request.

Author information

Authors and Affiliations



MT: conception and design of the study, managed the expert panel meetings which provided the input for the actual development of TREAT, data collection, data analysis and interpretation, wrote the draft article. LR: wrote the draft article, data analysis and interpretation. EL: conception and design of the study, data analysis, critical review of the manuscript. HK: conception and design of the study, critical review of the manuscript. PD: conception and design of the study, critical review of the manuscript. AB: conception and design of the study, critical review of the manuscript. MS: conception and design of the study, managed the expert panel meetings which provided the input for the actual development of TREAT, critical review of the manuscript. SC: conception and design of the study, data analysis and interpretation, critical review of the manuscript. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Lukas O. Roebroek.

Ethics declarations

Ethics approval and consent to participate

All participants signed an informed consent before participating in the study. We declare that the mental health experts of this study have ruled all participants mentally competent to ethically consent to their participation. The procedures of this study were in accordance with the declaration of Helsinki. This study has been approved by the Medical Ethics Committee of the University Medical Center Groningen.

Competing interests

The authors declare that they have no competing interests.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Additional information

Magda Tasma and Lukas Roebroek are shared first author.

Additional file

Additional file 1:

Clinical decision making in daily care - Staff version. (DOCX 41 kb)

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (, which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver ( applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Tasma, M., Roebroek, L.O., Liemburg, E.J. et al. The development and evaluation of a computerized decision aid for the treatment of psychotic disorders. BMC Psychiatry 18, 163 (2018).

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: