A surrogate measure for patient reported symptom remission in administrative data
BMC Psychiatry volume 21, Article number: 121 (2021)
In real-world pragmatic administrative databases, patient reported remission is often missing.
We evaluate if, in administrative data, five features of antidepressant use patterns can replace patient-reported symptom remission.
We re-examined data from Sequence Treatment Alternatives to Relieve Depression (STAR*D) study. Remission was measured using 50% reduction in Hamilton index. Pattern of antidepressant use was examined through five variables: (a) number of prior ineffective antidepressants, (b) duration of taking current antidepressant, (c) receiving therapeutic dose of the medication, and (d) switching to another medication, or (e) augmenting with another antidepressant. The likelihood ratio (LR) associated with each of these predictors was assessed in 90% of data (3329 cases) and evaluated in 10% of data (350 cases) set-aside for evaluation. The accuracy of predictions was calculated using Area under the Receiver Operating Curve (AROC).
Patients who took antidepressants for 14 weeks (LR = 2.007) were more likely to have symptom remission. Prior use of 3 antidepressants reduced the odds of remission (LR = 0.771). Patients who received antidepressants below therapeutic dose were 5 times less likely to experience remission (LR = 0.204). Antidepressant that were augment or switched, almost never led to remission (LR = 0.008, LR = 0.002 respectively). Patterns of antidepressant use accurately (AROC = 0.93) predicted symptom remission.
Within the first 100 days, antidepressants use patterns could serve as a surrogate measure for patient-reported remission of symptoms.
Well-designed observational studies can complement, and in some occasions replace, clinical trials of effectiveness of antidepressants [1, 2]. Observation studies rely on large sample size to detect side-effects and risks associated with taking antidepressants . These studies can show how effectiveness of antidepressants rely on patients’ multiple comorbidities . Observational studies can also facilitate the examination of antidepressants effectiveness subsets of patients, clarifying differences in reaction to antidepressants among nationality, ethnic, and racial minorities. Despite the obvious advantages of studying comparative effectiveness of antidepressants through observational studies, few such studies are done; in part, because the most crucial outcome of antidepressant use, i.e. patient-reported remission of symptoms, is not typically available in Electronic Health Records.
The effort to create surrogate measure for remission of symptoms is not new. Investigators have organized complex scoring systems to capture likelihood of remission in bipolar depression . Some studies use interaction between antidepressants and various other medications [6,7,8]. Many studies use antidepressant duration/discontinuation as a marker of effectiveness [9,10,11,12,13,14,15]. A number of investigators have suggested that effectiveness of antidepressants should be measured at the level of function and not symptom remission . By function, it is often understood to include employment and maintenance of family unit. In claims data, social functioning is rarely coded, but if coded these variables show as Z and V codes. In this paper, we rely on five features of antidepressant-use as a predictor of remission:
Duration of use,
Failure to reach therapeutic dose levels,
Switch from one antidepressant to another 
Treatment resistance as measured by prior use of multiple antidepressants .
This study reports how accurately these patterns of antidepressant use predict patient’s self-reported symptom remission.
This project was reviewed by George Mason University IRB and was considered exempt because it relied on de-identified and publicly available data.
Source of data
The Sequenced Treatment Alternatives to Relieve Depression (STAR*D) Study, is a public health clinical trial that is funded by the National Institute of Mental Health . The STAR*D study is the largest community-based effectiveness study of antidepressant use that also reports symptom remission. Pattern of antidepressant use is available until 14 weeks after prescription and therefore, long-term use cannot be evaluated.
Details of demographic and clinical features of the STAR*D participants have been published previously in multiple papers : 75.8% of participants were White, 17.6% Black, 13.0% were Hispanic, 41.7% were married, 26.5% were divorced, and 63.7% were female. The mean age was 40.8 years (standard deviation of 13.0 years). 62% of the participants were from psychiatric care settings. Depressive symptoms were moderate to severe. 75% of the patients had recurrent or chronic depression with mean length of depression of 15.5 years, with an average of 3.3 general medical comorbidities.
The STAR*D data were randomly divided into two sets: training and testing. Specifically, 90% of the data were used for training, i.e. estimating the parameters of the model; and the remaining 10% of data were used for testing the accuracy of the model.
Timing of variables
We focus on predicting symptom remission in the first 100 days after start of the antidepressant. This constitutes a short-term evaluation of remission, during a period when patients actively try different medications.
Calculation of likelihood ratio
The likelihood ratio (LR) calculates how many times symptom remission is more likely when a feature is present. It was calculated from the training data set using the following formula :
A ratio larger than 1 increases the odds of symptom remission. A ratio less than 1 decreases the odds of symptom remission.
Prediction of remission
The odds of remission was calculated as the prior odds times product of the likelihood ratios associated with the patient’s features:
For instance, consider a patient who has taken 2 prior unsuccessful antidepressants (LR = 0.771) and then takes a third antidepressant for 14 weeks (LR = 2.44). The prior odds in this example was calculated as 0.6. Then, the odds of remission is calculated as 0.6*0.771* 2.44 = 1.129, which corresponds to a probability of experiencing remission of 0.53.
Test of accuracy
In the 10% set aside validation data (treatment episodes of 350 cases), the accuracy of the prediction of remission, in the first 100 days, was calculated using the Area under the Receiver Operating Curve (AROC) . These curves are constructed from the sensitivity and specificity of the predictions at different cutoff levels. The AROC of 1 is perfect prediction and AROC of 0.5 indicates random prediction.
Figure 1 shows the likelihood ratios associated with duration of taking antidepressants. The longer the patient takes their medication, the more likely they would be to experience symptom remission. A likelihood ratio greater than 1 is observed only after 12 weeks of using the antidepressant, highlighting that remission does not tend to occur when the medication is taken for short duration. There are variations in the data (for example, after 14th weeks the likelihood ratio was lower than the value for 12 weeks). Across the entire data, the best line fitted to the data suggests that the longer one takes the antidepressant the more likely they will have remission.
Table 1 shows the LR of remission associated with different characteristics of antidepressant use, using the data in the training data set. Patients who switched to another antidepressant were unlikely to have experienced remission (LR=1/283) prior to the switch. It did not matter if patients switched or augmented their medication (LR = 0.008). In either case, few or no patients experienced remission. Patients who neither switched nor had an augmentation were nearly equally likely to experience remission (LR = 1.039). These data suggest that a switch or augmentation typically occurs after a prior medication has not been successful.
Patients who had not taken any antidepressants previously were most likely to experience remission. As the number of prior antidepressants increased, the likelihood ratio of remission decreased (Table 1). If the patient had taken 4 antidepressants, then he/she was 1.5 times less likely to experience remission (LR = 0.667).
Table 1 also shows the impact of reaching therapeutic dose levels at any time during the course of the treatment. Patients whose antidepressant met the minimum therapeutic level were nearly equally likely to experience remission or no remission (LR = 0.982). Patients whose medication did not meet the minimum therapeutic level were nearly five times less likely to experience remission (LR = 0.204).
For ease of understanding, the probability scores can be replaced with several simplified rules. We split each factor into a binary variable and scored various combination of these variables. We worked with clinicians to identify assignment of symptom remission based on aggregate probability of remission. These rules are described in Table 2. These rules suggest that only when anti-depressants are taken for at least 10 weeks, there is no switch or augmentation, and dose is above therapeutic limit, then remission is highly likely.
The accuracy of the model was evaluated in predicting remission for treatment episodes of 350 cases, not used in estimating the likelihood ratios. The overall accuracy of predictions can be seen in Fig. 2, which shows the sensitivity and specificity of predictions at different cutoff levels. The X-axis shows one minus specificity and the Y-axis shows the sensitivity of the predictions. The further the curve is from the diagonal line, the more accurate the prediction. The AROC associated with this curve was 0.93.
This study has several limitations. Antidepressant use patterns may be affected by cost or administrative reasons. For example, changes in insurance formularies can affect patterns of use of antidepressants. These types of changes may confound the initiation, continuation, switching, and/or termination of antidepressants. The current study also does not address the impact of changes in formulary.
This paper did not address long-term effectiveness of antidepressants. Further research is needed to extend the proposed measure of outcome to long-term use of antidepressants. The goal of long-term use of antidepressants, i.e. use beyond first 100 days, is to prevent relapse of depression. Therefore, for long-term effectiveness of antidepressants we propose to use days till (a) new diagnosis of depression, (b) intentional self-harm/suicide, (c) premature death, (d) certain side-effects and complications of use of antidepressants, e.g. falls, sleeping problems, sexual dysfunctions, bleeding [24, 25], (e) abandoning treatment against medical advice, and (f) psychiatric hospitalization.
Finally, it may not be reasonable to assume that what worked in STAR-D study may work for other settings. In day-to-day practice, the guides to dosing given in STAR*D does not exist. Any special rules for trying to prevent study dropout does not exist. The clinicians in STAR*D may have tried harder to accomplish remission because they were aware their effort was being studied.
Patterns of antidepressant use nearly perfectly (AROC=0.93) predicted symptom remission. Therefore, these patterns can be used as a surrogate measure of remission. Specifically, patients taking medications for less than 12 weeks, who take antidepressants below the therapeutic level, who have had more than 2 prior antidepressants that were not successful, who switched or augmented their antidepressants are less likely to experience symptom remission.
Given the level of experimentation that goes on within the first 100 days of use of antidepressants, it is not surprising that patterns of use predict remission. The study agrees with widely held intuition that short use of antidepressants is a clear indication of lack of benefit. Why continue with a medication, if it is not working?
This study also provides support for the intuition that when a medication is continued, sometimes there is remission and other times there is not. Some patients may continue with their medications, even when they do not experience remission of their symptoms. Other patients experience remission. By 14th week, if patients continue to use their antidepressant, they are twice more likely (LR = 2.007) to have experienced remission. While continuation does not guarantee remission, it increases the odds of remission.
This paper has shown that antidepressant use patterns are an accurate, although probabilistic, predictor of remission during the first 100 days of a depression episode. Future research can compare different methods of evaluating antidepressant outcomes. Of particular interest would be to contrast (1) patient reported remission, (2) remission as predicted from antidepressant use patterns, (3) patient’s ability to function at home/work, and (4) other outcomes of use of antidepressants.
Availability of data and materials
Data are publicly available from National Institute of Mental Health Data Archive at https://nda.nih.gov/edit_collection.html?id=2148.
Area under the Receiver Operating Curve
Sequence Treatment Alternatives to Relieve Depression study
Silverman SL. From randomized controlled trials to observational studies. Am J Med. 2009;122(2):114–20.
Benson K, Hartz AJ. A comparison of observational studies and randomized, controlled trials. N Engl J Med. 2000;342(25):1878–86.
Laporte S, Chapelle C, Caillet P, Beyens MN, Bellet F, Delavenne X, Mismetti P, Bertoletti L. Bleeding risk under selective serotonin reuptake inhibitor (SSRI) antidepressants: a meta-analysis of observational studies. Pharmacol Res. 2017;118:19–32.
Holvast F, van Hattem BA, Sinnige J, Schellevis F, Taxis K, Burger H, Verhaak PFM. Late-life depression and the association with multimorbidity and polypharmacy: a cross-sectional study. Fam Pract. 2017;34(5):539–45.
Perlis RH, Ostacher MJ, Patel JK, Marangell LB, Zhang H, Wisniewski SR, Ketter TA, Miklowitz DJ, Otto MW, Gyulai L, Reilly-Harrington NA, Nierenberg AA, Sachs GS, Thase ME. Predictors of recurrence in bipolar disorder: primary outcomes from the systematic treatment enhancement program for bipolar disorder. Am J Psychiatry. 2006;163(2):217–24.
Bauer M, Rush AJ, Ricken R, Pilhatsch M, Adli M. Algorithms for treatment of major depressive disorder: efficacy and cost-effectiveness. Pharmacopsychiatry. 2019;52(3):117–25.
Trivedi MH, Rush AJ, Crismon ML, Kashner TM, Toprac MG, Carmody TJ, Key T, Biggs MM, Shores-Wilson K, Witte B, Suppes T, Miller AL, Altshuler KZ, Shon SP. Clinical results for patients with major depressive disorder in the Texas medication algorithm project. Arch Gen Psychiatry. 2004;61(7):669–80.
Rush AJ, Crismon ML, Kashner TM, Toprac MG, Carmody TJ, Trivedi MH, Suppes T, Miller AL, Biggs MM, Shores-Wilson K, Witte BP, Shon SP, Rago WV, Altshuler KZ, TMAP Research Group. Texas medication algorithm project, phase 3 (TMAP-3): rationale and study design. J Clin Psychiatry. 2003;64(4):357–69.
Ten Have M, et al. Duration of major and minor depressive episodes and associated risk indicators in a psychiatric epidemiological cohort study of the general population. Acta Psychiatr Scand. 2017.
Maund E, Stuart B, Moore M, Dowrick C, Geraghty AWA, Dawson S, Kendrick T. Managing antidepressant discontinuation: a systematic review. Ann Fam Med. 2019;17(1):52–60.
Swanson SA, Hernandez-Diaz S, Palmsten K, Mogun H, Olfson M, Huybrechts KF. Methodological considerations in assessing the effectiveness of antidepressant medication continuation during pregnancy using administrative data. Pharmacoepidemiol Drug Saf. 2015;24(9):934–42.
Umeda T, Hayashi A, Fujimoto G, Piao Y, Matsui N, Tokita S. Medication adherence/persistence and demographics of Japanese dyslipidemia patients on statin-Ezetimibe as a separate pill combination lipid-lowering therapy - an observational pharmacy claims database study. Circ J. 2019;83(8):1689–97.
Öhlund L, Ott M, Oja S, Bergqvist M, Lundqvist R, Sandlund M, Salander Renberg E, Werneke U. Reasons for lithium discontinuation in men and women with bipolar disorder: a retrospective cohort study. BMC Psychiatry. 2018;18(1):37. https://doi.org/10.1186/s12888-018-1622-1.
Wu E, Greenberg P, Yang E, Yu A, Ben-Hamadi R, Erder MH. Comparison of treatment persistence, hospital utilization and costs among major depressive disorder geriatric patients treated with escitalopram versus other SSRI/SNRI antidepressants. Curr Med Res Opin. 2008;24(10):2805–13.
Locklear JC, Alemayehu B, Brody RS, Chavoshi S, Tunceli O, Kern D, Earley WR. Treatment patterns, healthcare resource utilization and costs in patients with bipolar disorder, newly treated with extended release or immediate release quetiapine fumarate using US healthcare administrative claims data. Clin Ther. 2013;35(12):1923–32.
Venditti LN, Arcelus A, Birnbaum H, Greenberg P, Barr CE, Rowland C, Williamson T. The impact of antidepressant use on social functioning: reboxetine versus fluoxetine. Int Clin Psychopharmacol. 2000;15(5):279–89.
Fleurence R, Williamson R, Jing Y, Kim E, Tran QV, Pikalov AS, Thase ME. A systematic review of augmentation strategies for patients with major depressive disorder. Psychopharmacol Bull. 2009;42(3):57–90.
Dold M, Kasper S. Evidence-based pharmacotherapy of treatment-resistant unipolar depression. Int J Psychiatry Clin Pract. 2017;21(1):13–23.
Connolly KR, Thase ME. If at first you don't succeed: a review of the evidence for antidepressant augmentation, combination and switching strategies. Drugs. 2011;71(1):43–64.
Zimmerman M, Balling C, Chelminski I, Dalrymple K. Understanding the severity of depression: which symptoms of depression are the best indicators of depression severity? Compr psychiatry. 2018;87:84–8.
Rush AJ, Fava M, Wisniewski SR, Lavori PW, Trivedi MH, Sackeim HA, Thase ME, Nierenberg AA, Quitkin FM, Kashner TM, Kupfer DJ, Rosenbaum JF, Alpert J, Stewart JW, McGrath PJ, Biggs MM, Shores-Wilson K, Lebowitz BD, Ritz L, Niederehe G, STAR*D investigators group. Sequenced treatment alternatives to relieve depression (STAR*D): rationale and design. Control Clin Trials. 2004;25(1):119–42.
Trivedi MH, Rush AJ, Wisniewski SR, Nierenberg AA, Warden D, Ritz L, Norquist G, Howland RH, Lebowitz B, McGrath PJ, Shores-Wilson K, Biggs MM, Balasubramani GK, Fava M, STAR*D study team. Evaluation of outcomes with citalopram for depression using measurement-based care in STAR*D: implications for clinical practice. Am J Psychiatry. 2006;163(1):28–40.
Alemi F. Big data: statistical analysis of electronic health record. Chicago: Health Administration Press; 2020.
Bet PM, Hugtenburg JG, Penninx BW, Hoogendijk WJ. Side effects of antidepressants during long-term use in a naturalistic setting. Eur Neuropsychopharmacol. 2013;23(11):1443–51.
Anderson HD, Pace WD, Libby AM, West DR, Valuck RJ. Rates of 5 common antidepressant side effects among new adult and adolescent cases of depression: a retrospective US claims study. Clin Ther. 2012;34(1):113–23.
This research was supported by grant #76786 from the Robert Wood Johnson Foundation.
Ethics approval and consent to participate
This research was reviewed and approved by George Mason University Institutional Review Board.
Consent for publication
All authors give consent for publication in the journal in an open format.
None of the authors have a conflicting interest to declare.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Alemi, F., Aljuaid, M., Durbha, N. et al. A surrogate measure for patient reported symptom remission in administrative data. BMC Psychiatry 21, 121 (2021). https://doi.org/10.1186/s12888-021-03133-1