Discriminant and concurrent validity of a simplified DSM-based structured diagnostic instrument for the assessment of autism spectrum disorders in youth and young adults
© Joshi et al; licensee BioMed Central Ltd. 2011
Received: 27 April 2011
Accepted: 30 December 2011
Published: 30 December 2011
To evaluate the concurrent and discriminant validity of a brief DSM-based structured diagnostic interview for referred individuals with autism spectrum disorders (ASDs).
To test concurrent validity, we assessed the structured interview's agreement in 123 youth with the expert clinician assessment and the Social Responsiveness Scale (SRS). Discriminant validity was examined using 1563 clinic-referred youth.
The structured diagnostic interview and SRS were highly sensitive indicators of the expert clinician assessment. Equally strong was the agreement between the structured interview and SRS. We found evidence for high specificity for the structured interview.
A simplified DSM-based ASD structured diagnostic interview could serve as a useful diagnostic aid in the assessment of subjects with ASDs in clinical and research settings.
Autism spectrum disorders (ASDs) comprise a group of neuropsychiatric disorders that include autistic disorder, Asperger's disorder, and pervasive developmental disorder not otherwise specified (PDD-NOS). They are distinguished from other psychiatric disorders by the presence of deficits in reciprocal social behavior, variously accompanied by deficits in communication, and/or repetitive or stereotyped behaviors. The DSM-III-R  and DSM-IV  have operationalized the required diagnostic criteria necessary for establishing diagnoses of ASDs based on the presence or absence of a set of categorical symptoms.
While a thorough evaluation by an expert clinician who has significant experience in specific diagnoses is considered the best method of diagnosing complex conditions such as ASDs, structured diagnostic interviews have been developed to help non-expert clinicians elicit the required information for these diagnoses. The most widely used structured interview tool for establishing a diagnosis of autism in the research setting is the Autism Diagnostic Interview-Revised (ADI-R). This interviewer requires specelized training in order to administer it, and the training to become proficient in its administration is expensive and time consuming. Additionally, the ADI-R takes at least 2 hours to complete making it of limited feasibility in clinical settings and in large population-based studies.
In contrast to the 93 questions and associated complex algorithms of the ADI-R, the DSM includes only 16 items in its diagnostic criteria for ASDs. Moreover, because the literature clearly indicates that most individuals with ASD have other comorbid conditions such as Attention-Defecit/Hyperactivity Disorder (ADHD) and mood disorders [3–7] and the ADI-R by itself does not assess other disorders that are pervasive in this population, another means for diagnosing other conditions is necessary and increases the length of time needed for a full assessment. This situation calls for the development of simplified instruments to aid in the assessment of ASDs in clinical and non-clinical settings.
Several attempts have been made to simplify the complexity of the assessment process for youth with ASDs. One such effort is development of the Social Responsiveness Scale (SRS), a paper and pencil instrument that can be completed by parents or teachers in 15-20 minutes. Constantino and colleagues [8, 12] demonstrated its concurrent and discriminant validity as a measure of ASDs. As part of these efforts, Constantino et al.  compared the SRS with the ADI-R in 61 child psychiatric patients. Correlations between SRS scores and ADI-R algorithm scores for DSM-IV criterion sets were on the order of 0.7. SRS scores were unrelated to I.Q. and exhibited inter-rater reliability on the order of 0.8. Though SRS is a valid quantitative measure of autistic traits, the instrument lacks the ability to distinguish autism from the spectrum (Asperger's disorder and PDD-NOS) in individuals with ASD . Despite the utility of the SRS as a screening tool for ASD, there continues to be a need to have a simplified DSM-based structured diagnostic interview module to aid in the diagnosis of individuals with ASD in clinical and research settings.
ASD symptoms for the structured interview, DSM-III-R and DSM-IV
ASD structured interview
DSM-III-R Autistic Disorder
DSM-IV Autistic Disorder (Bold=Asperger's Disorder)
A. THESE ARE SOME QUESTIONS ABOUT HOW YOUR SON/DAUGHTER RELATES TO OTHERS.
A. QUALITATIVE IMPAIRMENT IN RECIPROCAL SOCIAL INTERACTION
(1). QUALITATIVE IMPAIRMENT IN SOCIAL INTERACTION
1. Did s/he seem unusually unaware of the existence or feelings of others?
1. Marked lack of awareness of the existence of or feelings of others
(d) Lack of social or emotional reciprocity
2. Did s/he not come for comfort even when hurt, or did s/he seek comfort in an odd way
2. No or abnormal seeking of comfort at times of distress
3. Was s/he unable to imitate others when appropriate?
3. No or impaired imitation
4. Does s/he have difficulty playing cooperatively with other children?
4. No or abnormal social play
(c) A lack of spontaneous seeking to share enjoyment, interests, or achievements with other people
5. Is s/he uninterested in making peer friendships? Or if s/he is interested, does s/he seem to understand the conventions of social interaction?
5. Gross impairment in ability to make peer friendships
(b) Failure to develop peer relationships appropriate to developmental level
B. THESE ARE SOME QUESTIONS ABOUT HOW YOUR SON/DAUGHTER COMMUNICATES OR PLAYS WITH OTHERS.
B. QUALITATIVE IMPAIRMENT IN VERBAL AND NONVERBAL COMMUNICATION, AND IN IMAGINATIVE ACTIVITY
(2). QUALITATIVE IMPAIRMENTS IN COMMUNICATION
1. Is s/he unable to communicate?
1. No mode of communication, such as communicative babbling, facial expression, gesture, mime, or spoken language
(a) Delay in, or total lack of, the development of spoken language (not accompanied by an attempt to compensate through alternative modes of communication such as gesture or mime)
2. Does s/he avoid looking at people or avoid greeting people? If no, ask: Does s/he ignore people around her/him, dislike being held?
2. Markedly abnormal nonverbal communication, as in the use of eye-to-eye gaze, facial expression, body posture, or gestures to initiate or modulate social interaction
(1) (a) Marked impairment in the use of multiple nonverbal behaviors such as eye-to-eye gaze, facial expression, body postures, and gestures to regulate social interaction
3. Is s/he uninterested in imaginative activities or stories?
3. Absence of imaginative activity, such as playacting of adult roles, fantasy characters, or animals; lack of interest in stories about imaginary events
(d) Lack of varied, spontaneous make-believe play or social imitative play appropriate to developmental level
4. When s/he speaks does her/his tone seem odd?
4, Marked abnormalities in the production of speech, including volume, pitch, stress, rate, rhythm, and intonation
5. Did s/he repeat words or phrases s/he has just heard, in place of responding to what was said? Did s/he often use the wrong pronouns to refer to her/himself or others, or refer to him/herself in the third person, as "he wants a remarks cracker?"
5. Marked abnormalities in the form or content of speech, including stereotyped and repetitive use of speech; use of "you" when "I" is meant; idiosyncratic use of words or phrases; or frequent irrelevant remarks
(c) Stereotyped and repetitive use of language or idiosyncratic language
6. Did s/he seldom, if ever, start a conversation with someone else, even if s/he might talk to her/himself?
6. Marked impairment in the ability to initiate or sustain a conversation with others, despite adequate speech
(b) In individuals with adequate speech, marked impairment in the ability to initiate or sustain a conversation with others
C. THESE ARE QUESTIONS ABOUT SON/DAUGHTER' S ACTIVITIES OR INTERESTS
C. MARKEDLY RESTRICTED REPERTOIRE OF ACTIVITIES AND INTERESTS
(3) R ESTRICTED REPETITIVE AND STEREOTYPED PATTERNS OF BEHAVIOR, INTERESTS, AND ACTIVITIES
1. Did s/he ever have any repetitive patterns of behavior such as hand movements, clapping or twirling?
1. Stereotyped body movements, e.g., hand-flicking or -twisting, spinning, head-banging, complex whole-body movements
(c) Stereotyped and repetitive motor mannerisms
2. Did s/he ever have any prolonged attachments to certain objects, either holding them or staring at them, or lining them up in a repetitive pattern?
2. Persistent preoccupation with parts of objects or attachment to unusual objects
(d) Persistent preoccupation with parts of objects
3. Did s/he ever get unusually upset if there were even small changes in where things were placed in the house?
3. Marked distress over changes in trivial aspects of environment
4. Or get upset when there are changes in daily routine?
4. Unreasonable insistence on following routines in precise detail
(b) Apparently inflexible adherence to specific, nonfunctional routines or rituals
5. Does s/he have an extremely restricted range of interests, or a preoccupation with one very narrow interest
5. Markedly restricted range of interests and a preoccupation with one narrow interest
(a) Encompassing preoccupation with one or more stereotyped and restricted patterns of interest that is abnormal either in intensity or focus
ASD subjects were youth, ages 4 to 23 years of age, consecutively referred to a specialized program for the treatment of ASDs at a university-affiliated hospital. The diagnosis of ASD was established by a comprehensive psychiatric evaluation conducted by an board-certified psychiatrist experienced in evaluating ASD (GJ). The psychiatric diagnostic interview was conducted with the subject and caretaker (usually parent/s) and incorporated information from multiple sources when available (psychiatric records, schools, social services). Based on this clinical evaluation, all ASD subjects met Diagnostic and Statistical Manual of Mental Disorders - Fourth Edition (DSM-IV) diagnostic criteria for autistic disorder, Asperger's disorder, or PDD-NOS.
Psychiatric comparison participants were derived from consecutive referrals to a pediatric psychopharmacology program at a major academic center from 1991 to 2008. Children were referred for psychiatric evaluation and psychopharmacological intervention for behavioral and emotional difficulties and not for evaluation of any specific disorder. There was no selection bias based on social class or insurance restrictions. We included subjects if they met diagnostic criteria for ADHD on a structured diagnostic interview (Schedule for Affective Disorder and Schizophrenia for School-Age Children Epidemiologic Version [K-SADS-E]) . The status of ASD in the ADHD control participants was assessed by the DSM based structured interview for ASD (see below). The structured diagnostic interview was completed by highly trained and supervised psychometricians from interviews with the parent, usually the mother. We computed kappa coefficients of agreement between these raters and experienced board certified child and adult psychiatrists and licensed clinical psychologists. Based on 500 assessments, the median kappa coefficient was 0.98. Kappa coefficients for ADHD was 0.88. Before final diagnostic assignments were made, information derived from these interviews was reviewed blindly by a committee of expert clinicians that included board certified child and adult psychiatrists and experienced licensed psychologists. We estimated the reliability of the diagnostic review process by computing kappa coefficients of agreement for clinician reviewers. For these diagnoses, the median reliability between individual clinicians and the review committee assigned diagnoses was 0.87. Kappa coefficients for ADHD was 1.0.
For each ASD subject, a parent also completed The Social Responsiveness Scale (SRS) a 65 item rating scale that measures the severity of autism spectrum symptoms including elements of reciprocal social behaviors (39 items), social use of language (6 items), and behaviors characteristic of children with autism spectrum disorders (20 items). Each item on the scale is rated on a Likert scale from "0" (never true) to "3" (almost always true). The psychometric properties of the SRS have been well established [11–14].
DSM-based structured interview for ASD
To assess ASDs we developed DSM-based diagnostic criterion into an interview format (see Table 1) using the parent as the informant. Diagnostic assessment of ASDs by this interview required a lifelong severe and pervasive deficit in development of reciprocal social interaction, communication, and restricted patterns of behavior. ASD subjects were defined as subjects meeting criteria for autistic disorder or PDD-NOS. To be given the diagnosis of autistic disorder, the participant had to meet DSM-III-R diagnostic criteria of eight out of sixteen symptoms with at least two symptoms from each of the three aforementioned domains of PDD. A diagnosis of PDD-NOS was received if more than two of the required symptoms were met with at least one symptom present from each of the three domains of PDD. This DSM-III-R based structured interview for ASD was developed prior to the release of DSM-IV criterion for ASD. This interview for ASD was administered to all the participants in this study. As data collection for the psychiatric comparison group in this study preceded the advent of DSM-IV, in order to maintain consistency in assessment the DSM-III-R criterion was retained beyond the release of DSM-IV. This DSM-based structured interview for ASD is added as a module to K-SADS-E and is administered by the trained interviewer in similar manner as the structured interview. All questions in the structured interview are asked in yes/no format. If the interviewee positively endorses a question, interviewers have specific follow-up questions they are required to ask. These questions include ages a symptom began/ended, if such statements have been true in the past month, and specific examples to elaborate on responses to the initial probes. Responses to follow-up questions help to determine whether each criterion is met. Supporting the interrater reliability of this diagnostic interview, the kappa coefficient of agreement for ASDs between the raters and experienced board certified child and adult psychiatrists and licensed clinical psychologists was 0.90. For ASDs, the reliability between an individual clinician and the review committee assigned diagnoses was 0.88. Table 1 also addresses the correlation between DSM-III-R criteria and DSM-IV criteria that clearly indicate that our DSM-based interview sufficiently covers both versions of the DSM.
Full scale IQ
Subjects with ASDs were assessed with the WASI . Psychiatric comparison subjects were assessed with the WISC-R (N = 464) , WISC-3 (N = 97) , or the WASI (N = 77) . The study was approved by the Institutional Review Board and in all cases parents gave written informed consent for participation.
Conditional probabilities were calculated by determining how many clinically diagnosed ASD subjects and psychiatric comparison subjects met ASD criteria using the structured interview ASD module. We also determined how many ASD subjects had SRS scores of 60 or above. Then we examined the relationship between the structured interview diagnosis of ASD and scores on the SRS using two-sample t-tests. We randomly excluded 30% of the females from the original pool of the psychiatric comparison group so that both groups had similar gender ratios.
Demographics and IQ scores
ASD (N = 123)
ADHD (N = 1563)
10.9 ± 4.3
10.3 ± 3.3
Full Scale IQ
96.7 ± 18.4b
103.9 ± 15.1c
Eleven percent (172/1563) of the ADHD group met structured interview criteria for ASDs on the DSM-based structured interview for ASDs. Therefore, a conservative estimate of the specificity of the DSM-based structured interview for ASD was 89%. Positive predictive value was 40% (116/288), and negative predictive value was 99.8% (1391/1398). In the ADHD group, subjects with a positive ASD diagnosis from the DSM-based structured interview were significantly younger than subjects without an ASD diagnosis (9.6 ± 3.5 versus 10.4 ± 3.3, z = -2.92, p = 0.003). Age was not significantly associated with the DSM-based structured interview diagnosis of ASD in the ASD group (z = 0.05, p = 0.96). In both of the groups, there was no relationship between meeting criteria for a structured interview diagnosis of ASDs and full scale IQ (ASD, z = 0.60, p = 0.54; ADHD, z = 0.51, p = 0.61). In the ASD group, subjects with a previous diagnosis of ASD were significantly more likely to receive a structured interview diagnosis of ASDs compared to subjects who not been previously diagnosed with ASD (previously diagnosed = 109/112, 97%; not previously diagnosed = 7/11, 64%; χ2(1) = 21.18, p < 0.001). Likewise, the clinician diagnosis was significantly associated with the structured interview diagnosis of ASD (autism = 100% received structured interview diagnosis of ASD; Asperger's disorder = 91%; PDD-NOS = 81%; χ2(2) = 13.88, p = 0.001).
The main purpose of the present study has been to evaluate the concurrent and discriminant validity of a simplified DSM-based structured diagnostic interview for the assessment of ASDs in a clinical setting. The present study reports excellent agreement of the DSM-based ASD structured diagnostic instrument with a gold standard expert clinician diagnosis of ASD based on a DSM-IV clinical assessment through detailed interviews with the patient and the parent. Results also showed excellent sensitivity and specificity when comparisons were made between subjects with ASD and subjects with ADHD. These results indicate that our DSM-based structured diagnostic interview for ASD can be a useful and cost-effective standardized assessment instrument for reliably identifying ASD in clinical and research settings. Many clinics and research settings employ diagnostic structured interviews for screening a broad range of psychiatric disorders but these structured interviews lack measures to evaluate ASD. Therefore, this DSM based structured diagnostic interview - that is administered in a similar manner as structured interviews - would complement methods that are often used and can be easily added to the diagnostic interviews. This may improve the efficiency of the assessment, as it is included with screening of other psychiatric conditions.
Our design provides a reasonable estimate of sensitivity (i.e., the probability of our structured interview correctly identifying ASD cases). Remarkably, the sensitivity of the structured interview was extremely high, 94%. Equally remarkable is the finding of a 95% sensitivity of an SRS t-score of 60 or higher (accepted cut-off for a screen of ASD) as an indicator of the gold standard diagnosis. Consistent with these findings, 97% of subjects with our DSM based structured interview diagnosis of ASD also had an SRS t-score in the clinical range (≥ 60). If replicated, these findings would support the utility of a simple to use structured diagnostic instrument based on the defining items for ASD in the DSM-IV to help identify youth with ASD.
Results from our analysis show that the correspondence between our DSM-based structured interview for ASD with the expert clinician assessment was unrelated to the IQ of the subjects in both clinical samples. In addition, our structured interview allowed for the diagnosis of subjects with subthreshold disorders and different definitions of ASDs such as Asperger's Disorder or PDD-NOS, where the ADI-R algorithm score for social impairment may fall below the published clinical cutoff. Thus, our DSM-based structured interview for ASD may be useful and accurate for the assessment of ASD individuals across different cognitive and developmental levels with full and subsyndromal manifestations of these disorders.
Our results must be interpreted in the context of some methodological limitations. Since subjects in this study were referred for ASDs, our results may not generalize to other clinical and non-clinical settings. Because our sample consisted largely of Caucasian subjects, we do not know whether our results will generalize to other ethnic groups. The SRS is validated for youth ages 4 to 18 years but 4% (5/123) of our ASD sample was older than 18 years and received the scale. Although our DSM-based structured interview for ASD was DSM-III-R based, there have been very few changes between the DSM-III-R definitions of ASD and those in DSM-IV (see Table 1). Moreover, we documented a very high correspondence between the DSM-IV based clinician diagnosis of ASD and this instrument. Just as the current version of the structured interview is able to capture both DSM-III-R and DSM-IV diagnostic criteria, a revised version will capture DSM-V measures. As currently proposed, DSM-V criteria for Autism Spectrum Disorders are more narrow and unidimentional. As such, the DSM-based structured interview presented in this paper incorporates the criteria proposed in DSM-V. Coding criteria could also be altered in the future to encompass the changes proposed in the diagnostic criteria for ASD in DSM-V.
Despite these considerations, our results document the utility of a DSM-based, simple to use and administer, relatively brief structured diagnostic instrument to aid in the identification of youth with ASDs in the clinical setting. If confirmed, these findings would suggest that our DSM-based structured diagnostic instrument for ASD could serve as a rapid and cost-effective assessment instrument to help identify cases likely to meet clinical criteria for ASDs in clinical and non-clinical settings.
This work was supported in part by a grant to the Alan and Lorraine Bressler Center for study of Autism Spectrum Disorders and the Norma Fine Pediatric Psychopharmacology Fellowship Fund.
- American Psychiatric Association: DSM-III-R. 1987, Washington D.C.: American Psychiatric Association
- American Psychiatric Association: Diagnostic and Statistical Manual of Mental Disorders. 1994, Washington, D.C.: American Psychiatric Association, 4
- de Bruin EI, Ferdinand RF, Meester S, de Nijs PF, Verheij F: High rates of psychiatric co-morbidity in PDD-NOS. J Autism Dev Disord. 2007, 37 (5): 877-886. 10.1007/s10803-006-0215-x.View ArticlePubMed
- Leyfer OT, Folstein SE, Bacalman S, Davis NO, Dinh E, Morgan J, Tager-Flusberg H, Lainhart JE: Comorbid psychiatric disorders in children with autism: interview development and rates of disorders. J Autism Dev Disord. 2006, 36 (7): 849-861. 10.1007/s10803-006-0123-0.View ArticlePubMed
- Wozniak J, Biederman J: Mania in children with PDD. J Am Acad Child Adolesc Psychiatry. 1997, 36 (12): 1646-1647.PubMed
- Green AI, Tohen M, Patel JK, Banov M, DuRand C, Berman I, Chang H, Zarate C, Posener J, Lee H, et al: Clozapine in the treatment of refractory psychotic mania. American Journal of Psychiatry. 2000, 157 (6): 982-986. 10.1176/appi.ajp.157.6.982.View ArticlePubMed
- Muris P, Steerneman P, Merckelbach H, Holdrinet I, Meesters C: Comorbid anxiety symptoms in children with pervasive developmental disorders. J Anxiety Disord. 1998, 12 (4): 387-393. 10.1016/S0887-6185(98)00022-X.View ArticlePubMed
- Constantino JN, Davis SA, Todd RD, Schindler MK, Gross MM, Brophy SL, Metzger LM, Shoushtari CS, Splinter R, Reich W: Validation of a brief quantitative measure of autistic traits: comparison of the social responsiveness scale with the autism diagnostic interview-revised. J Autism Dev Disord. 2003, 33 (4): 427-433. 10.1023/A:1025014929212.View ArticlePubMed
- Orvaschel H: Schedule for Affective Disorder and Schizophrenia for School-Age Children Epidemiologic Version. 1994, Ft. Lauderdale: Nova Southeastern University, Center for Psychological Studies, 5
- Constantino JN: The Social Responsiveness Scale. 2002, Los Angeles: Western Psychological Services
- Constantino JN, Todd RD: Genetic structure of reciprocal social behavior. Am J Psychiatry. 2000, 157 (12): 2043-2045. 10.1176/appi.ajp.157.12.2043.View ArticlePubMed
- Constantino JN, Przybeck T, Friesen D, Todd RD: Reciprocal social behavior in children with and without pervasive developmental disorders. J Dev Behav Pediatr. 2000, 21 (1): 2-11. 10.1097/00004703-200002000-00002.View ArticlePubMed
- Constantino JN, Todd RD: Autistic traits in the general population: a twin study. Arch Gen Psychiatry. 2003, 60 (5): 524-530. 10.1001/archpsyc.60.5.524.View ArticlePubMed
- Constantino JN, Todd RD: Intergenerational transmission of subthreshold autistic traits in the general population. Biol Psychiatry. 2005, 57 (6): 655-660. 10.1016/j.biopsych.2004.12.014.View ArticlePubMed
- Wechsler D: Wechsler Abbreviated Scale of Intelligence (WASI). 1999, San Antonio, Tx: The Psychological Corporation, 4
- Wechsler D: Manual for the Wechsler Intelligence Scale for Children-Revised. 1974, New York: The Psychological Corporation
- Wechsler D: Manual for the Wechsler Intelligence Scale for Children. 1991, San Antonio: The Psychological Corporation, Harcourt Brace Jovanovich, Inc, 3
- The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1471-244X/11/204/prepub
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.