Skip to main content

A surrogate measure for patient reported symptom remission in administrative data

Abstract

Background

In real-world pragmatic administrative databases, patient reported remission is often missing.

Objective

We evaluate if, in administrative data, five features of antidepressant use patterns can replace patient-reported symptom remission.

Method

We re-examined data from Sequence Treatment Alternatives to Relieve Depression (STAR*D) study. Remission was measured using 50% reduction in Hamilton index. Pattern of antidepressant use was examined through five variables: (a) number of prior ineffective antidepressants, (b) duration of taking current antidepressant, (c) receiving therapeutic dose of the medication, and (d) switching to another medication, or (e) augmenting with another antidepressant. The likelihood ratio (LR) associated with each of these predictors was assessed in 90% of data (3329 cases) and evaluated in 10% of data (350 cases) set-aside for evaluation. The accuracy of predictions was calculated using Area under the Receiver Operating Curve (AROC).

Results

Patients who took antidepressants for 14 weeks (LR = 2.007) were more likely to have symptom remission. Prior use of 3 antidepressants reduced the odds of remission (LR = 0.771). Patients who received antidepressants below therapeutic dose were 5 times less likely to experience remission (LR = 0.204). Antidepressant that were augment or switched, almost never led to remission (LR = 0.008, LR = 0.002 respectively). Patterns of antidepressant use accurately (AROC = 0.93) predicted symptom remission.

Conclusion

Within the first 100 days, antidepressants use patterns could serve as a surrogate measure for patient-reported remission of symptoms.

Peer Review reports

Introduction

Well-designed observational studies can complement, and in some occasions replace, clinical trials of effectiveness of antidepressants [1, 2]. Observation studies rely on large sample size to detect side-effects and risks associated with taking antidepressants [3]. These studies can show how effectiveness of antidepressants rely on patients’ multiple comorbidities [4]. Observational studies can also facilitate the examination of antidepressants effectiveness subsets of patients, clarifying differences in reaction to antidepressants among nationality, ethnic, and racial minorities. Despite the obvious advantages of studying comparative effectiveness of antidepressants through observational studies, few such studies are done; in part, because the most crucial outcome of antidepressant use, i.e. patient-reported remission of symptoms, is not typically available in Electronic Health Records.

The effort to create surrogate measure for remission of symptoms is not new. Investigators have organized complex scoring systems to capture likelihood of remission in bipolar depression [5]. Some studies use interaction between antidepressants and various other medications [6,7,8]. Many studies use antidepressant duration/discontinuation as a marker of effectiveness [9,10,11,12,13,14,15]. A number of investigators have suggested that effectiveness of antidepressants should be measured at the level of function and not symptom remission [16]. By function, it is often understood to include employment and maintenance of family unit. In claims data, social functioning is rarely coded, but if coded these variables show as Z and V codes. In this paper, we rely on five features of antidepressant-use as a predictor of remission:

  1. (a)

    Duration of use,

  2. (b)

    Failure to reach therapeutic dose levels,

  3. (c)

    Augmentation of antidepressants with other medications [17, 18]

  4. (d)

    Switch from one antidepressant to another [19]

  5. (e)

    Treatment resistance as measured by prior use of multiple antidepressants [20].

This study reports how accurately these patterns of antidepressant use predict patient’s self-reported symptom remission.

Methods

This project was reviewed by George Mason University IRB and was considered exempt because it relied on de-identified and publicly available data.

Source of data

The Sequenced Treatment Alternatives to Relieve Depression (STAR*D) Study, is a public health clinical trial that is funded by the National Institute of Mental Health [21]. The STAR*D study is the largest community-based effectiveness study of antidepressant use that also reports symptom remission. Pattern of antidepressant use is available until 14 weeks after prescription and therefore, long-term use cannot be evaluated.

Details of demographic and clinical features of the STAR*D participants have been published previously in multiple papers [22]: 75.8% of participants were White, 17.6% Black, 13.0% were Hispanic, 41.7% were married, 26.5% were divorced, and 63.7% were female. The mean age was 40.8 years (standard deviation of 13.0 years). 62% of the participants were from psychiatric care settings. Depressive symptoms were moderate to severe. 75% of the patients had recurrent or chronic depression with mean length of depression of 15.5 years, with an average of 3.3 general medical comorbidities.

The STAR*D data were randomly divided into two sets: training and testing. Specifically, 90% of the data were used for training, i.e. estimating the parameters of the model; and the remaining 10% of data were used for testing the accuracy of the model.

Timing of variables

We focus on predicting symptom remission in the first 100 days after start of the antidepressant. This constitutes a short-term evaluation of remission, during a period when patients actively try different medications.

Calculation of likelihood ratio

The likelihood ratio (LR) calculates how many times symptom remission is more likely when a feature is present. It was calculated from the training data set using the following formula [23]:

$$ {\mathrm{LR}}_{Feature}=\frac{\mathrm{Prevalence}\ \mathrm{of}\ \mathrm{feature}\ \mathrm{for}\ \mathrm{patients}\ \mathrm{with}\ \mathrm{remission}}{\mathrm{Prevalence}\ \mathrm{of}\ \mathrm{feature}\ \mathrm{for}\ \mathrm{patients}\ \mathrm{with}\mathrm{out}\ \mathrm{remission}} $$

A ratio larger than 1 increases the odds of symptom remission. A ratio less than 1 decreases the odds of symptom remission.

Prediction of remission

The odds of remission was calculated as the prior odds times product of the likelihood ratios associated with the patient’s features:

$$ \mathrm{Odds}\ \mathrm{of}\ \mathrm{Remission}=\mathrm{Prior}\ \mathrm{Odds}\ast {\prod}_{\begin{array}{c}\mathrm{Patien}{\mathrm{t}}^{\prime}\mathrm{s}\\ {}\mathrm{Features}\end{array}}{\mathrm{LR}}_{\mathrm{Feature}} $$

For instance, consider a patient who has taken 2 prior unsuccessful antidepressants (LR = 0.771) and then takes a third antidepressant for 14 weeks (LR = 2.44). The prior odds in this example was calculated as 0.6. Then, the odds of remission is calculated as 0.6*0.771* 2.44 = 1.129, which corresponds to a probability of experiencing remission of 0.53.

Test of accuracy

In the 10% set aside validation data (treatment episodes of 350 cases), the accuracy of the prediction of remission, in the first 100 days, was calculated using the Area under the Receiver Operating Curve (AROC) [22]. These curves are constructed from the sensitivity and specificity of the predictions at different cutoff levels. The AROC of 1 is perfect prediction and AROC of 0.5 indicates random prediction.

Results

Figure 1 shows the likelihood ratios associated with duration of taking antidepressants. The longer the patient takes their medication, the more likely they would be to experience symptom remission. A likelihood ratio greater than 1 is observed only after 12 weeks of using the antidepressant, highlighting that remission does not tend to occur when the medication is taken for short duration. There are variations in the data (for example, after 14th weeks the likelihood ratio was lower than the value for 12 weeks). Across the entire data, the best line fitted to the data suggests that the longer one takes the antidepressant the more likely they will have remission.

Fig. 1
figure1

Likelihood Ratio of Remission and Duration of Medication

Table 1 shows the LR of remission associated with different characteristics of antidepressant use, using the data in the training data set. Patients who switched to another antidepressant were unlikely to have experienced remission (LR=1/283) prior to the switch. It did not matter if patients switched or augmented their medication (LR = 0.008). In either case, few or no patients experienced remission. Patients who neither switched nor had an augmentation were nearly equally likely to experience remission (LR = 1.039). These data suggest that a switch or augmentation typically occurs after a prior medication has not been successful.

Table 1 Symptom Remission Given Characteristics of Antidepressant Use. (Calculated from Remission Rate in Treatment Episodes of 3329 Training Cases)

Patients who had not taken any antidepressants previously were most likely to experience remission. As the number of prior antidepressants increased, the likelihood ratio of remission decreased (Table 1). If the patient had taken 4 antidepressants, then he/she was 1.5 times less likely to experience remission (LR = 0.667).

Table 1 also shows the impact of reaching therapeutic dose levels at any time during the course of the treatment. Patients whose antidepressant met the minimum therapeutic level were nearly equally likely to experience remission or no remission (LR = 0.982). Patients whose medication did not meet the minimum therapeutic level were nearly five times less likely to experience remission (LR = 0.204).

For ease of understanding, the probability scores can be replaced with several simplified rules. We split each factor into a binary variable and scored various combination of these variables. We worked with clinicians to identify assignment of symptom remission based on aggregate probability of remission. These rules are described in Table 2. These rules suggest that only when anti-depressants are taken for at least 10 weeks, there is no switch or augmentation, and dose is above therapeutic limit, then remission is highly likely.

Table 2 Rules for Classification of Symptom Remission from Features of Antidepressant Use

The accuracy of the model was evaluated in predicting remission for treatment episodes of 350 cases, not used in estimating the likelihood ratios. The overall accuracy of predictions can be seen in Fig. 2, which shows the sensitivity and specificity of predictions at different cutoff levels. The X-axis shows one minus specificity and the Y-axis shows the sensitivity of the predictions. The further the curve is from the diagonal line, the more accurate the prediction. The AROC associated with this curve was 0.93.

Fig. 2
figure2

Receiver Operating Curve for Predicting Remission. (Predictions Were Made for Treatment Episodes of 350 Set-Aside Evaluation Cases)

Limitations

This study has several limitations. Antidepressant use patterns may be affected by cost or administrative reasons. For example, changes in insurance formularies can affect patterns of use of antidepressants. These types of changes may confound the initiation, continuation, switching, and/or termination of antidepressants. The current study also does not address the impact of changes in formulary.

This paper did not address long-term effectiveness of antidepressants. Further research is needed to extend the proposed measure of outcome to long-term use of antidepressants. The goal of long-term use of antidepressants, i.e. use beyond first 100 days, is to prevent relapse of depression. Therefore, for long-term effectiveness of antidepressants we propose to use days till (a) new diagnosis of depression, (b) intentional self-harm/suicide, (c) premature death, (d) certain side-effects and complications of use of antidepressants, e.g. falls, sleeping problems, sexual dysfunctions, bleeding [24, 25], (e) abandoning treatment against medical advice, and (f) psychiatric hospitalization.

Finally, it may not be reasonable to assume that what worked in STAR-D study may work for other settings. In day-to-day practice, the guides to dosing given in STAR*D does not exist. Any special rules for trying to prevent study dropout does not exist. The clinicians in STAR*D may have tried harder to accomplish remission because they were aware their effort was being studied.

Discussion

Patterns of antidepressant use nearly perfectly (AROC=0.93) predicted symptom remission. Therefore, these patterns can be used as a surrogate measure of remission. Specifically, patients taking medications for less than 12 weeks, who take antidepressants below the therapeutic level, who have had more than 2 prior antidepressants that were not successful, who switched or augmented their antidepressants are less likely to experience symptom remission.

Given the level of experimentation that goes on within the first 100 days of use of antidepressants, it is not surprising that patterns of use predict remission. The study agrees with widely held intuition that short use of antidepressants is a clear indication of lack of benefit. Why continue with a medication, if it is not working?

This study also provides support for the intuition that when a medication is continued, sometimes there is remission and other times there is not. Some patients may continue with their medications, even when they do not experience remission of their symptoms. Other patients experience remission. By 14th week, if patients continue to use their antidepressant, they are twice more likely (LR = 2.007) to have experienced remission. While continuation does not guarantee remission, it increases the odds of remission.

This paper has shown that antidepressant use patterns are an accurate, although probabilistic, predictor of remission during the first 100 days of a depression episode. Future research can compare different methods of evaluating antidepressant outcomes. Of particular interest would be to contrast (1) patient reported remission, (2) remission as predicted from antidepressant use patterns, (3) patient’s ability to function at home/work, and (4) other outcomes of use of antidepressants.

Availability of data and materials

Data are publicly available from National Institute of Mental Health Data Archive at https://nda.nih.gov/edit_collection.html?id=2148.

Abbreviations

AROC:

Area under the Receiver Operating Curve

LR:

Likelihood Ratio

STAR*D:

Sequence Treatment Alternatives to Relieve Depression study

References

  1. 1.

    Silverman SL. From randomized controlled trials to observational studies. Am J Med. 2009;122(2):114–20.

    Article  Google Scholar 

  2. 2.

    Benson K, Hartz AJ. A comparison of observational studies and randomized, controlled trials. N Engl J Med. 2000;342(25):1878–86.

    CAS  Article  Google Scholar 

  3. 3.

    Laporte S, Chapelle C, Caillet P, Beyens MN, Bellet F, Delavenne X, Mismetti P, Bertoletti L. Bleeding risk under selective serotonin reuptake inhibitor (SSRI) antidepressants: a meta-analysis of observational studies. Pharmacol Res. 2017;118:19–32.

    CAS  Article  Google Scholar 

  4. 4.

    Holvast F, van Hattem BA, Sinnige J, Schellevis F, Taxis K, Burger H, Verhaak PFM. Late-life depression and the association with multimorbidity and polypharmacy: a cross-sectional study. Fam Pract. 2017;34(5):539–45.

    Article  Google Scholar 

  5. 5.

    Perlis RH, Ostacher MJ, Patel JK, Marangell LB, Zhang H, Wisniewski SR, Ketter TA, Miklowitz DJ, Otto MW, Gyulai L, Reilly-Harrington NA, Nierenberg AA, Sachs GS, Thase ME. Predictors of recurrence in bipolar disorder: primary outcomes from the systematic treatment enhancement program for bipolar disorder. Am J Psychiatry. 2006;163(2):217–24.

    Article  Google Scholar 

  6. 6.

    Bauer M, Rush AJ, Ricken R, Pilhatsch M, Adli M. Algorithms for treatment of major depressive disorder: efficacy and cost-effectiveness. Pharmacopsychiatry. 2019;52(3):117–25.

    Article  Google Scholar 

  7. 7.

    Trivedi MH, Rush AJ, Crismon ML, Kashner TM, Toprac MG, Carmody TJ, Key T, Biggs MM, Shores-Wilson K, Witte B, Suppes T, Miller AL, Altshuler KZ, Shon SP. Clinical results for patients with major depressive disorder in the Texas medication algorithm project. Arch Gen Psychiatry. 2004;61(7):669–80.

    Article  Google Scholar 

  8. 8.

    Rush AJ, Crismon ML, Kashner TM, Toprac MG, Carmody TJ, Trivedi MH, Suppes T, Miller AL, Biggs MM, Shores-Wilson K, Witte BP, Shon SP, Rago WV, Altshuler KZ, TMAP Research Group. Texas medication algorithm project, phase 3 (TMAP-3): rationale and study design. J Clin Psychiatry. 2003;64(4):357–69.

    Article  Google Scholar 

  9. 9.

    Ten Have M, et al. Duration of major and minor depressive episodes and associated risk indicators in a psychiatric epidemiological cohort study of the general population. Acta Psychiatr Scand. 2017.

  10. 10.

    Maund E, Stuart B, Moore M, Dowrick C, Geraghty AWA, Dawson S, Kendrick T. Managing antidepressant discontinuation: a systematic review. Ann Fam Med. 2019;17(1):52–60.

    Article  Google Scholar 

  11. 11.

    Swanson SA, Hernandez-Diaz S, Palmsten K, Mogun H, Olfson M, Huybrechts KF. Methodological considerations in assessing the effectiveness of antidepressant medication continuation during pregnancy using administrative data. Pharmacoepidemiol Drug Saf. 2015;24(9):934–42.

    Article  Google Scholar 

  12. 12.

    Umeda T, Hayashi A, Fujimoto G, Piao Y, Matsui N, Tokita S. Medication adherence/persistence and demographics of Japanese dyslipidemia patients on statin-Ezetimibe as a separate pill combination lipid-lowering therapy - an observational pharmacy claims database study. Circ J. 2019;83(8):1689–97.

    Article  Google Scholar 

  13. 13.

    Öhlund L, Ott M, Oja S, Bergqvist M, Lundqvist R, Sandlund M, Salander Renberg E, Werneke U. Reasons for lithium discontinuation in men and women with bipolar disorder: a retrospective cohort study. BMC Psychiatry. 2018;18(1):37. https://doi.org/10.1186/s12888-018-1622-1.

    Article  PubMed  PubMed Central  Google Scholar 

  14. 14.

    Wu E, Greenberg P, Yang E, Yu A, Ben-Hamadi R, Erder MH. Comparison of treatment persistence, hospital utilization and costs among major depressive disorder geriatric patients treated with escitalopram versus other SSRI/SNRI antidepressants. Curr Med Res Opin. 2008;24(10):2805–13.

    CAS  Article  Google Scholar 

  15. 15.

    Locklear JC, Alemayehu B, Brody RS, Chavoshi S, Tunceli O, Kern D, Earley WR. Treatment patterns, healthcare resource utilization and costs in patients with bipolar disorder, newly treated with extended release or immediate release quetiapine fumarate using US healthcare administrative claims data. Clin Ther. 2013;35(12):1923–32.

    Article  Google Scholar 

  16. 16.

    Venditti LN, Arcelus A, Birnbaum H, Greenberg P, Barr CE, Rowland C, Williamson T. The impact of antidepressant use on social functioning: reboxetine versus fluoxetine. Int Clin Psychopharmacol. 2000;15(5):279–89.

    CAS  Article  Google Scholar 

  17. 17.

    Fleurence R, Williamson R, Jing Y, Kim E, Tran QV, Pikalov AS, Thase ME. A systematic review of augmentation strategies for patients with major depressive disorder. Psychopharmacol Bull. 2009;42(3):57–90.

    PubMed  Google Scholar 

  18. 18.

    Dold M, Kasper S. Evidence-based pharmacotherapy of treatment-resistant unipolar depression. Int J Psychiatry Clin Pract. 2017;21(1):13–23.

    CAS  Article  Google Scholar 

  19. 19.

    Connolly KR, Thase ME. If at first you don't succeed: a review of the evidence for antidepressant augmentation, combination and switching strategies. Drugs. 2011;71(1):43–64.

    CAS  Article  Google Scholar 

  20. 20.

    Zimmerman M, Balling C, Chelminski I, Dalrymple K. Understanding the severity of depression: which symptoms of depression are the best indicators of depression severity? Compr psychiatry. 2018;87:84–8.

    Article  Google Scholar 

  21. 21.

    Rush AJ, Fava M, Wisniewski SR, Lavori PW, Trivedi MH, Sackeim HA, Thase ME, Nierenberg AA, Quitkin FM, Kashner TM, Kupfer DJ, Rosenbaum JF, Alpert J, Stewart JW, McGrath PJ, Biggs MM, Shores-Wilson K, Lebowitz BD, Ritz L, Niederehe G, STAR*D investigators group. Sequenced treatment alternatives to relieve depression (STAR*D): rationale and design. Control Clin Trials. 2004;25(1):119–42.

    Article  Google Scholar 

  22. 22.

    Trivedi MH, Rush AJ, Wisniewski SR, Nierenberg AA, Warden D, Ritz L, Norquist G, Howland RH, Lebowitz B, McGrath PJ, Shores-Wilson K, Biggs MM, Balasubramani GK, Fava M, STAR*D study team. Evaluation of outcomes with citalopram for depression using measurement-based care in STAR*D: implications for clinical practice. Am J Psychiatry. 2006;163(1):28–40.

    Article  Google Scholar 

  23. 23.

    Alemi F. Big data: statistical analysis of electronic health record. Chicago: Health Administration Press; 2020.

    Google Scholar 

  24. 24.

    Bet PM, Hugtenburg JG, Penninx BW, Hoogendijk WJ. Side effects of antidepressants during long-term use in a naturalistic setting. Eur Neuropsychopharmacol. 2013;23(11):1443–51.

    CAS  Article  Google Scholar 

  25. 25.

    Anderson HD, Pace WD, Libby AM, West DR, Valuck RJ. Rates of 5 common antidepressant side effects among new adult and adolescent cases of depression: a retrospective US claims study. Clin Ther. 2012;34(1):113–23.

    CAS  Article  Google Scholar 

Download references

Acknowledgements

None.

Funding

This research was supported by grant #76786 from the Robert Wood Johnson Foundation.

Author information

Affiliations

Authors

Contributions

FA organized the study, received the funding, participated in the analysis. MA and ND analyzed the data. MY, HM, LGS, and AAN provided clinical advice on interpretation of findings. All authors participated in writing and revision of the report. The author(s) read and approved the final manuscript.

Corresponding author

Correspondence to Farrokh Alemi.

Ethics declarations

Ethics approval and consent to participate

This research was reviewed and approved by George Mason University Institutional Review Board.

Consent for publication

All authors give consent for publication in the journal in an open format.

Competing interests

None of the authors have a conflicting interest to declare.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Alemi, F., Aljuaid, M., Durbha, N. et al. A surrogate measure for patient reported symptom remission in administrative data. BMC Psychiatry 21, 121 (2021). https://doi.org/10.1186/s12888-021-03133-1

Download citation

Keywords

  • Prior medication use
  • Medication augmentation
  • Medication switch
  • Therapeutic dose
  • Antidepressant effectiveness
  • Symptom remission
  • STAR*D