Validation of online psychometric instruments for common mental health disorders: a systematic review

Transdiagnostic online instruments

Seven instruments assessed both depressive and anxiety symptoms, or anxiety symptoms that apply to several disorders, such as the Beck Anxiety Inventory (BAI). These can be roughly divided in short instruments that screen for disorders, e.g. the Web Screening Questionnaire (WSQ) [18] and the Web-Based Depression and Anxiety Test (WB-DAT) [8], and scales that assess symptom severity, e.g. the Hospital Anxiety and Depression Scale (HADS) [19] and the Depression Anxiety Stress Scales (DASS) [20]. The short screening questionnaires had poor to adequate criterion validity for screening individual disorders [8, 18, 21]. Of the symptom severity scales, the HADS was investigated in 5 studies [19, 22–25]. These 5 studies showed a fair to good internal consistency. The online HADS is the only instrument we found that was investigated among several patient populations [19, 23, 24]. Although the factor structure may be different from how the measure was designed [19, 23], there is mounting evidence that support adequate validity of the online HADS.

Online assessment of depression

Our review includes 29 instruments that measure depressive symptoms. These consist of 22 instruments that measure depression alone and 7 transdiagnostic instruments. The 22 studies on instruments for depression generally reported recruiting their samples from the general population. Five studies investigated instruments for depression among patient populations [3, 6, 26–28], each investigating a different instrument.

The full version of the CES-D has been evaluated in 6 studies [5, 29–33], and 5 characteristics were each reported by at least 2 studies (Table 2). Moreover, all 6 studies recruited their samples among non-patients, so the results can be considered complementary. The internal consistency was investigated in 5 of these studies, reporting a Cronbach’s alpha of .89 to .93. Factor analysis showed that the CES-D consists of 2, 3 or 4 factors [32, 33]. The 2-factor solution was among an English speaking population, the 3-factor solution among a Spanish speaking and the 4-factor solution among a Chinese speaking population [32, 33]. Adequate psychometric characteristics were found for the CES-D regarding equivalence of mean scores with the paper version [31, 33], convergent validity [5, 30] and criterion validity [5, 30]. One study [33] conducted a full measurement invariance analysis using confirmatory factor analysis, comparing paper and online formats, and found only a negligible difference in the latent mean score of one factor. Overall, it can be concluded that the online CES-D has good psychometric characteristics among non-patient populations, and that a start has been made to investigate its intercultural validity.

Another commonly investigated instrument was the MADRS-S [3, 4, 34–37]. Five of these studies reported Cronbach’s alpha, which is adequate to excellent (.73 to .90, Table 2) [3, 4, 34–36]. Thorndike and colleagues [37] found that the scale consists of 3 factors. Four studies found that the mean score of the MADRS-S does not differ significantly between the online and the paper version [3, 4, 35, 36].

Online assessment of GAD

The GAD-7 and two shorter versions were studied among a sample recruited from the general population [38]. The scale showed promising internal consistency, convergent validity and predictive validity. The psychometrics of the GAD-7 were similar among a population of people with hearing loss [6].

Online assessment of panic disorder and agoraphobia

Internet interventions for PD/A, such as self-help courses, have been relatively extensively researched. Therefore, Austin and colleagues [39] and Carlbring and colleagues [4] studied the online questionnaires usually employed for such research. They focussed on equivalence of mean scores with paper versions of the same instruments. This equivalence could generally be assumed due to high correlations, but the study of Carlbring [4] found that online versions yield significantly lower mean scores for the Body Sensations Questionnaire (BSQ) and Agoraphobic Cognitions Questionnaire (ACQ) and higher scores for the Mobility Inventory (MI) subscale Alone. Finally, an agoraphobia screening item augmented with images was found to have adequate criterion validity (AUC .73) [7]. All these studies recruited their samples from the general population.

Online assessment of social phobia

Two studies [3, 40] independently investigated the equivalence between online and paper versions of the online versions of the Social Interaction Anxiety Scale (SIAS) and Social Phobia Scale (SPS). Both did not find a difference between formats in mean score, but the factor structure did differ between formats [40], indicating that scores cannot be compared across formats. Adequate to good internal consistency of these scales has also been found in three studies [3, 40, 41], and adequate convergent validity of the SIAS in two [40, 41]. Lindner and colleagues revised item 14 of the SIAS, because the original item only applied to heterosexual people. This change did not alter the internal consistency or convergent validity of the scale [41]. The study of Hedman and colleagues [3] recruited people classified with social phobia, but more research among patient groups is recommended.

Online assessment of specific phobia

Two of the transdiagnostic screening measures [18, 21] included specific phobia. These showed poor criterion validity for specific phobia. One instrument, the Flight Anxiety Situations Questionnaire (FAS), has been studied for aviophobia [42]. This study showed near perfect criterion validity (AUC .99). Considering aviophobia is only one of many different specific phobias, much more development is needed in this area.

Online assessment of OCD

Four instruments for OCD have been studied, all in the US and among the general population [43–45]. Each instrument was studied only once. Williams and colleagues [45] investigated differential item functioning between black and white Americans, finding significant differences for the Padua Inventory (PI). Next to these 4 instruments, the WSQ [18] and the CIDI-SF [21] also screen for OCD.

Online assessment of PTSD

Like instruments for OCD, 4 instruments for PTSD have been studied, all in the US and among the general population [31, 46–48]. The transdiagnostic WSQ [18] also screens for PTSD. One additional study investigated an instrument for perinatal PTSD [49]. Miller and colleagues [47] checked the factor structure of their measure for PTSD (National Stressful Events Survey) using item-response theory. The factor structure was confirmed, but the items of the instrument may cover too narrow a range of the latent factors.

Online assessment of worry and stress

The PSWQ, assessing worry, was studied twice [20, 50]. These studies found slightly differing values for internal consistency (.73 and .88). We found one study on an instrument that assesses stress [51].

Online assessment of suicidal ideation and self-harm

We found one study on an instrument that assesses self-harm. [52] This study used Rasch analysis to further confirm the factors of the Inventory of Statements About Self-injury (ISAS), obtained by factor analysis, and their unidimensionality. Furthermore, we found two single-item measures for suicidal ideation, being item 9 of the BDI-II and item 9 of the MADRS-S [36]. Item 9 of the online BDI-II yielded lower scores than item 9 of the paper version of the BDI-II [36]. The WSQ [18] also contains an item that screens for suicidal ideation, but the validity of this item was not investigated (also see [53]).