Setting
The East Harlem Health Outreach Partnership (EHHOP) is a student-run and faculty-supervised clinic affiliated with the Icahn School of Medicine at Mount Sinai in New York, NY. Established in 2004, EHHOP provides free primary care to East Harlem adults (22 years and older) who are unable to obtain health insurance, most often because they have not met legal residency requirements. In 2018, 12% of East Harlem adults reported not having insurance [18], and a 2017 study estimated that there were at least 14,000 immigrants living in East Harlem who did not meet residency requirements [19]. Compared to most other NYC neighborhoods, East Harlem has higher rates of unemployment, violent crime, and premature death and a rate of psychiatric hospitalizations that is three times the NYC average [18]. Demographically, 50% of East Harlem residents identify as Hispanic and 30% as black [18].
Approach to the evaluation and treatment of psychiatric disorders
The EHHOP Mental Health Clinic (E-MHC) is a co-habiting clinic that accepts patients who receive primary care at the main medical clinic of EHHOP. Interdisciplinary management is key to its success, as student clinicians in the primary care clinic and E-MHC co-manage patients with a high prevalence of complex medical disease and psychiatric illness.
At initial intake to EHHOP and at least once annually, patients are screened for depressive and anxiety disorders using the Patient Health Questionnaire-9 (PHQ-9) [20] and the Generalized Anxiety Disorder-7 Scale (GAD-7) [21], respectively. Patients with positive screening results on either measure or who otherwise express mental health concerns are referred to the E-MHC for further evaluation and treatment as necessary. Clinical services provided to E-MHC patients include psychiatric assessment, medication management, non-specific supportive counseling, and individual psychotherapy conducted by supervised medical student trainees. New patients are seen at least once monthly for medication management and more frequently if they are receiving psychotherapeutic interventions. After stabilization, a minority of patients are transitioned to bi-monthly or quarterly follow-up visits.
Fourth-year psychiatric residents, volunteer psychiatrists, and supervising clinical psychologists oversee the services provided by second to fourth year medical students or MD-PhD students who have completed the first year of medical school. Following all E-MHC patient appointments, the student trainees present their patient to a supervising psychiatrist or fourth-year resident in psychiatry. Initial diagnoses are based upon unstructured interviews by the student, who then finalizes the diagnostic formulation with the supervising psychiatrist or resident. Supervisors review the patient’s status, formulate a treatment plan with the student, and provide additional mentoring in outpatient psychiatry. After these discussions, both the student and the supervisor meet with the patient to answer questions, review the assessment and treatment plan, and ensure that there are no safety concerns.
Patients expressing suicidality at any time are given a more thorough risk assessment by the supervisor; if needed, patients are taken to a nearby emergency room for continued monitoring and stabilization. As needed, on call psychiatry and medical faculty supervise trainees who triage phone calls; faculty provide necessary navigation of care and communication with emergency room and inpatient teams.
Psychotropic medications are prescribed by the supervising psychiatrist, and patients receive their medications with no out-of-pocket costs either at a Mount Sinai Hospital pharmacy or on-site immediately after their appointments [16]. Of note, there is a limited formulary of medications stratified by cost on a web-based application that providers consult when prescribing medications. In between the Saturdays on which the E-MHC is open, first and second year medical and graduate students manage the clinic’s schedule and coordinate follow-up visits and appointment reminders for all patients.
Determination of demographic and clinical characteristics of the patient population
Age, race/ethnicity, and sex (male or female) were identified by review of patients’ electronic medical records from January 1st, 2009 to March 1st, 2020. Psychiatric diagnoses were collected from the patient’s charts and confirmed in provider notes; patients could have multiple diagnoses if they were concomitantly diagnosed or if different diagnoses were listed throughout the course of treatment. Psychiatric diagnoses collected included major depressive disorder, persistent depressive disorder, seasonal affective disorder, depression not otherwise specified (NOS), generalized anxiety disorder, panic disorder, social anxiety disorder, somatic symptom disorder, anxiety not otherwise specific (NOS), adjustment disorder, post-traumatic stress disorder, borderline personality disorder, substance use disorders, and persistent complex bereavement disorder. We also collected information about current and past sexual assault and intimate partner violence (SA/IPV) based upon review of provider notes. There were no pre-defined hypotheses tested, and only summary statistics are presented.
Evaluation of mental healthcare service performance
We evaluated the quality of mental healthcare services at the E-MHC using the Healthcare Effectiveness Data and Information Set (HEDIS) performance metrics established by the National Committee on Quality Assurance. Many previous studies of clinical care performance utilize the HEDIS metrics, as they are empirically derived and objectively defined measures with specific criteria designed to operationalize each aspect of healthcare performance [22]. The New York State (NYS) Department of Health (DOH) mandates reporting of HEDIS metric data from all major managed care plans in the state and makes the data publicly available through the NYS DOH website and various published reports [23,24,25]. Managed care plans include insurance plans provided through Medicaid, Preferred Partner Organizations (PPOs), and Health Maintenance Organizations (HMOs). We refer the reader to the NYS DOH published reports for more information on the characteristics of each of these plan types and the specific plans included [24, 25]. We selected HEDIS behavioral healthcare metrics based upon those that were relevant to the clinical services provided by the E-MHC in the year 2019 and that could be calculated using the metric’s definition and the availability of patient data. We were able to compare our performance on metrics for “optimal provider contacts for treatment of depressive disorders,” “receipt of effective acute- and continuation-phase antidepressant treatment,” multiple metrics relating to smoking cessation intervention, and “follow-up care after ED visits for alcohol and other drug dependencies (AOD).” The definitions and criteria for having met each metric are as follows:
Optimal provider contacts for treatment of depressive disorders
Defined as the patient attending three or more follow-up visits within the first 3 months after an initial diagnosis of a depressive disorder; at least one of these visits must be with the provider overseeing medication-management [14]. We defined the date of initial diagnosis of a depressive disorder as the date listed in the electronic medical record visit note corresponding to the intake assessment visit. We confirmed attendance to follow-up visits by cross-referencing all patient encounters listed with the corresponding visits notes.
Receipt of effective acute-phase and continuation-phase antidepressant treatment
These two metrics were defined according to the HEDIS criteria outlined under “Antidepressant Medication Management (AMM),” which applies to adults ages 18 years or older with a diagnosed depressive disorder who were newly treated with an antidepressant medication [22]. “Effective Acute Phase Treatment” is defined as those who actively took a prescribed antidepressant medication for at least 12 weeks, and “Effective Continuation Phase Treatment” is defined as those who actively took a prescribed antidepressant medication for at least 6 months [22]. We determined whether patients were actively taking their antidepressant medications by reviewing all visit notes within the specified time period following initiation of the medication; patients were considered to have met each of the metric criteria if the visit notes explicitly stated that the patient self-reported full medication adherence on greater than 80% of visits within the corresponding time periods.
Smoking cessation
We evaluated three HEDIS metrics related to smoking cessation. “Advising Smokers and Tobacco Users to Quit” is defined as whether patients who were current smokers or tobacco users received any cessation advice during the past year. We first determined which patients were current smokers or tobacco users through chart review and then read through each visit note from the prior year to see if there was any indication that the provider gave cessation advice. “Discussing Cessation Medications” refers to current smokers or tobacco users to whom the provider expressed a recommendation to consider cessation medications (e.g., varenicline, bupropion, etc.) in the past year, which we assessed via chart review. Finally, “Discussing Cessation Strategies” refers to current smokers or tobacco users who discussed or were provided with information on cessation strategies or behavioral methods during the past year. We assessed whether patients had received such information via chart review and used a liberal definition of “cessation strategies or behavioral methods” to include informal advice during a visit on some of the different techniques or approaches that can be used to quit smoking, the provider indicating that they gave the patient print-outs with information on cessation approaches, and any more formal discussion of cessation strategies including provision of behavioral techniques and/or any psychotherapeutic interventions [22].
Follow-up care after ED visits for alcohol or other drug dependencies
This is defined as patients with an established substance use disorder to alcohol and/or other substances who had a visit to the emergency department (ED) related to their substance use and who attended a follow-up visit within a specified period of time after discharge from the ED or hospital. There are two rates reported, one for attendance to a follow-up visit within 30 days of ED visit and one for attendance to a follow-up visit within 7 days of an ED visit. We determined which patients had a diagnosed substance use disorder and then ascertained both ED visits and attendance to follow-up visits via chart review [22].
We hypothesized that the E-MHC would perform at levels non-inferior to those of NYS clinics grouped by insurance type. To conduct hypothesis-testing, we first retrieved data collected and published by the NYS DOH [23,24,25]. For the effective antidepressant medication management acute- and continuation-phase metrics, we also included our previously published data [15] in the comparisons to determine if the E-MHC’s performance improved over time. Due to the small sample size in our study and the unbalanced group sizes, Fisher exact tests were used to quantify the likelihood of patients having met the specific metric criteria between the E-MHC in 2019 and those from each of the comparator groups. Results are reported as odds ratios with 95% confidence intervals (CIs) and were considered significant if p < 0.05; if results were non-significant, they were operationalized to mean that the E-MHC performance was non-inferior to that of the comparator group.
Assessment of factors associated with clinical outcomes
We longitudinally assessed depressive and anxious symptoms using the PHQ-9 and GAD-7, respectively, which were available from patients’ charts as part of routine care and symptom-monitoring. We first defined baseline symptom scores as either the score upon referral to the E-MHC or the score(s) reported at the initial assessment visit. Patients who had neither and had no scores reported within the first month of treatment were excluded from the analyses. End-point scores were defined as those achieved at the last recorded visit for the patient within the study period. We only included those who had at least mild PHQ-9/GAD-7 symptoms at baseline, defined as a score of 5 or greater on both scales [20, 21]. For all statistical tests, major depressive disorder, depression NOS, persistent depressive disorder, seasonal affective disorder, and persistent complex bereavement disorder were combined into a single diagnostic category of “depression.” Similarly, generalized anxiety disorder, panic disorder, social anxiety disorder, and somatic symptom disorder were combined into a composite diagnostic category of “anxiety.” Below, we describe the hypothesis-testing procedures conducted:
-
(1)
We first sought to evaluate the extent to which patients’ depressive and anxious symptoms changed over the course of their treatment in the E-MHC. We hypothesized that end-point scores on both PHQ-9 and GAD-7 would be significantly lower than baseline scores. We used within-subjects paired t-tests for all patients included and within groups of patients categorized by psychiatric condition treated.
-
(2)
We then endeavored to model patients’ depressive and anxious symptom severity overtime throughout the course of treatment and ascertain factors accounting for differences in symptom severity. We tested the hypothesis that the number of treatment sessions received by patients in the E-MHC would be positively associated with the magnitude of symptom improvement. To do so, we used each patient’s repeated, longitudinal PHQ-9 and GAD-7 scores throughout treatment and generated linear mixed effects models to determine whether treatment session number (independent variable) was associated with symptom severity score (raw score on the PHQ-9 or GAD-7) as the dependent variable. Session number was treated as the fixed effect, and patient ID was treated as the random effect. Subsequently, we sought to determine whether the inclusion of additional demographic or clinical variables improved the extent to which session number predicted symptom score; variables of interest included age, sex, baseline PHQ-9 score, baseline GAD-7 score, whether the patient had more than one psychiatric diagnosis, the number of psychiatric diagnoses at baseline, and whether the patient was listed as having (Y/N) depression, anxiety, adjustment disorder, alcohol use disorder, post-traumatic stress disorder, and/or intimate partner violence. Candidate models were evaluated using all-subsets regression with the leaps package in R. Bayesian Information Criterion (BIC) scores, a measure of information explained by each model that penalizes for overfitting [26], were used for variable-selection.
-
(3)
Finally, we sought to determine what factors, if any, were associated with the probability that patients achieved a clinically significant improvement in symptoms by the end of treatment. To do so, we employed the current consensus definition of a “clinically significant improvement” (CSI) as a reduction in symptom severity of 50% or more [26, 27]. We hypothesized that higher treatment session number would be positively associated with the likelihood of attaining CSI. Subsequently, we evaluated multi-factorial models that included various combinations of the demographic and clinical variables considered in the evaluation of longitudinal symptom severity models mentioned above. To test these hypotheses, we generated logistic regression models with the dependent variable as whether the patient achieved a CSI in the PHQ-9/GAD-7 scores (Y/N). Candidate models were selected using all subsets regression with the bestglm() function of the bestglm package [28]. The model with the best BIC score was selected. Finally, we graphically illustrated model predictions using the predict() function in R.
Evaluation of patients’ feedback
We created a custom Patient Feedback Survey based upon frequently asked questions in the research literature on scales measuring patient satisfaction with mental healthcare services [29]. All questions were asked on a 5-point scale with possible answers of “Strongly Disagree,” “Disagree,” “Neutral,” “Agree,” and “Strongly Agree.” All feedback surveys were collected at the end of patient visits, and some patients completed the survey more than once over the course of their overall treatment in the E-MHC. A copy of the Patient Feedback Survey is provided in Supplementary File 1. We did not have any pre-defined hypotheses and provide only descriptive statistics on the results for each survey question.