Promoting early autism detection and intervention in underserved communities: study protocol for a pragmatic trial using a stepped-wedge design
BMC Psychiatry volume 19, Article number: 169 (2019)
Despite the known benefits of early, specialized intervention for toddlers with Autism Spectrum Disorder (ASD), access to such intervention remains limited. This pragmatic trial examines a novel healthcare delivery model (Screen-Refer-Treat [SRT]), which capitalizes upon existing health care and early intervention (EI) infrastructure to increase community capacity for ASD detection and treatment before age 3, when it is likely to have the greatest impact. This model comprises three components: (1) universal use of Stage 1 ASD screening by primary care providers (PCPs) at 18-month well-child visits (i.e., Screen); (2) immediate referral of positive screens to a community-based EI program (i.e., Refer); and (3) provision of an inexpensive, evidence-based ASD-specialized treatment by EI providers, after verifying ASD risk with a Stage 2 screen (i.e., Treat). This paper describes our research design and the initial successes, challenges, and adaptations made during the early implementation phase.
A stepped-wedge cluster RCT was used to implement the SRT model sequentially in four diverse Washington State counties (“clusters”). Counties are randomly assigned to the time of receipt of the SRT intervention, which comprises training workshops and technical assistance focused on the use of evidence-based ASD screening and intervention tools. Separate cohorts of families with toddlers (16–35 months old) with and without ASD concerns are recruited before and after the SRT intervention from participating PCP practices and EI programs. PCPs and EI providers complete measures on their screening, referral, and intervention practices before and after the SRT intervention. Each family cohort completes surveys about their well-being, parenting efficacy, health care satisfaction, and toddler’s social-communicative behaviors.
This trial is the first of its kind to work simultaneously with two service delivery systems with the goal of improving early detection and treatment for ASD. Our approach was successful in attaining buy-in from PCPs and EI providers, building and maintaining partnerships with providers, and achieving high levels of retention and survey completion. Fostering provider engagement and problem-solving issues together as partners were integral to overcoming the main challenges. Numerous lessons have been learned thus far, which have applicability for implementation researchers in ASD and those in other fields.
The registration number for this trial is NCT02409303 and it was posted on ClinicalTrials.gov on April 6, 2015.
Autism Spectrum Disorder (ASD) is a complex neurodevelopmental disorder characterized by severe impairment in two broad domains: social communication and behavioral rigidity . The prevalence of ASD has increased steadily over the years and is now estimated to affect 1 in 59 children in the United States (U.S.), with children from Hispanic backgrounds identified at lower rates and later ages due to inadequate detection . Although there is currently no known cure for ASD, early identification and specialized intervention have led to significant improvements in social, language, cognitive, and behavioral functioning [3,4,5,6,7]. To promote early detection and optimize child outcomes, the American Academy of Pediatrics (AAP) published ASD practice guidelines that include conducting universal ASD-specific screening with a standardized tool at 18 and 24 months of age . However, guideline compliance is low , with primary care providers (PCPs) reporting numerous barriers to screening, including insufficient time during well-child visits, work flow challenges, and reimbursement issues [10,11,12,13].
The prevalent model for ASD care requires a formal diagnosis from a qualified professional to obtain access to ASD-specialized intervention. Most communities have insufficient numbers of professionals with expertise in diagnosing or treating ASD; thus, many toddlers with ASD fail to receive appropriately specialized intervention during the birth-to-three years, when its impact might be greatest [14, 15]. Concerned parents may encounter delays of up to 2 years before receiving an ASD diagnosis and/or specialized services [16,17,18]. This service vacuum not only creates stress for concerned parents, but also provides a disincentive for PCPs to conduct early ASD screening [9, 16, 19]. In this paper, we describe an alternative service delivery model and a pragmatic trial we are conducting to evaluate the model.
Designing a new service delivery model for ASD: the screen-refer-treat model
The Screen-Refer-Treat (SRT) model employs state-of-the-art developments in ASD detection and intervention and capitalizes on a readily accessible service delivery infrastructure . The goal of the SRT model is to use a preventive intervention approach and mitigate emerging ASD behaviors during a critical period of development. This model introduces a significant shift in the conceptualization and implementation of ASD services by: (1) offering an ASD-specialized intervention to toddlers when ASD is first suspected, rather than waiting until a formal ASD diagnosis is conferred; (2) involving two integral and interrelated parts of the service delivery system—primary care and early intervention (EI)—to increase care coordination (See Fig. 1); and (3) addressing obstacles associated with early detection and intervention through the use of technology to expedite screening. Two stages of screening are used: Stage 1 (designed for primary care settings), and Stage 2 (designed for referral settings) .
The SRT model comprises three components: PCPs conduct universal Stage 1 ASD screening at the 18-month well-child visit (Screen); PCPs immediately refer toddlers who screen positive to a community-based EI program (Refer); and EI providers initiate ASD treatment (Treat) after verifying ASD risk with a Stage 2 screen.
PCPs screen children at 18-month well-child checks using the Modified Checklist for Autism in Toddlers-Revised, with Follow Up (M-CHAT-R/F) [22, 23]. The M-CHAT-R/F is a well-validated Stage 1 parent-report screening tool for detecting ASD in population-based settings. It comprises two components: a 20-item behavior checklist completed by parents; and follow-up questions conducted as an interview if the child screens positive on the checklist. In typical clinical practice, however, time constraints often prevent PCPs from administering the follow-up interview questions [9, 10], which can lead to elevated rates of false positive results [22, 24]. To circumvent this challenge, we developed a web-based version of the M-CHAT-R/F [25, 26] that automatically triggers the appropriate follow-up questions for parents to complete if the checklist score indicates that the child is “at risk”. Scoring is automated, and results are delivered electronically to the PCP.
PCPs immediately refer toddlers with a positive M-CHAT-R/F to a community-based EI program. These programs are federally-funded under the Individuals with Disabilities Education Act (IDEA)  to serve infants and toddlers under 36 months old who have delays or disabilities, and are available in all communities at no cost to families.
EI providers assess the need for ASD-specialized treatment using the Screening Tool for Autism in Toddlers (STAT) [28,29,30], a 12-item, interactive Stage 2 screen that is completed in 20 min and scored in real time. For children screening positive on the STAT, EI providers deliver an evidence-based, ASD-specialized behavioral intervention, Reciprocal Imitation Training (RIT) [31, 32]. RIT uses a naturalistic behavioral approach and a play-based context to teach imitation skills, which are a core deficit area for young children with ASD. Because imitation is a pivotal skill, RIT has also led to concurrent improvements in broader social-communication skills [31, 32].
This pragmatic trial was designed to test the effectiveness of the SRT model in improving both system-level and family-level outcomes. At the system level, our aims are to: (1) increase the number of toddlers receiving ASD screening at their 18-month well-child visit; (2) increase the number of toddlers with positive screens who receive evidence-based behavioral intervention prior to age 3; (3) reduce the length of time between parents’ expressed concerns about ASD and the child’s receipt of ASD-specialized intervention; and (4) reduce disparities in identification of ASD for Hispanic families, given the magnitude of the delays they experience relative to other underrepresented populations . At the family level, the SRT project aims to: (1) improve parents’ well-being, parenting efficacy, parenting stress, and health care satisfaction; and (2) improve children’s social-communication skills. This paper describes how the SRT trial was designed and is being conducted, highlighting successes and challenges we have encountered during trial implementation.
The SRT service delivery model was initiated in four underserved counties in geographically distinct areas of Washington (WA) State, using a pragmatic trial framework and a stepped-wedge cluster randomized controlled design (RCT) . The research hub was centralized at the University of Washington, where IRB approval was obtained prior to the commencement of study activities and for all protocol amendments. This paper reports on research activities and data collection that occurred during the first 22 months of this project, prior to implementation of SRT training activities. During this “Pre-SRT” period, two baseline Pre-SRT surveys have been collected from providers, and enrollment of the Pre-SRT cohort of families has been completed.
In contrast to explanatory trials, which test how an intervention works under optimal conditions, pragmatic trials are designed to evaluate the effectiveness of an intervention in routine practice conditions . Consistent with the pragmatic approach , SRT recruits community providers and the families whom they serve. All individual components of the SRT intervention have been established as efficacious and ethical and they are organized around usual clinic practice, allowing for flexibility in procedures, methods, and staffing. Pragmatic components of the SRT trial are summarized in Table 1.
Stepped-wedge cluster RCT design
After a period of baseline data collection in Year 1, each county (“cluster”) was randomly assigned (using random permutation/shuffling by the biostatistician) to one of four time periods during Year 2 when providers received their training in the SRT components (see Fig. 2). Our decision to employ the stepped-wedge design was based on several considerations. First, early feedback from providers revealed that their motivation to participate over a 5-year study period was contingent on their receipt of the SRT training. Second, this approach removes some of the logistical constraints associated with implementing a complex multi-level healthcare system intervention in multiple locations, by allowing us to stagger training activities for each county. Third, this design allows for concurrent comparisons between counties exposed to the SRT model and those not yet exposed; a simple pre-post comparison may be confounded by policy or system changes occurring between the pre-intervention and post-implementation periods. Randomization was at the county level (cluster) rather than at the organizational level within each county to minimize cross-contamination (i.e., providers already trained in one organization exposing the yet-to-be trained providers in another organization to the SRT components).
Changes in PCP and EI provider practices are examined using a within-subjects approach, by comparing service delivery practices before and after SRT intervention. Parent experiences are examined using a between-subjects approach, by comparing a sample of parents recruited prior to SRT intervention (Pre-SRT cohort) to a separate sample of parents recruited after SRT intervention (Post-SRT cohort); parents were not aware of the study design and, as such, did not know what cohort they were in.
Recruitment entailed first identifying interested counties, then identifying PCP practices and EI programs with interested providers, and finally recruiting parent/toddler dyads from the caseloads of the participating PCP practices and EI programs.
Counties were selected for participation based on: (1) demographic diversity; (2) the presence of local champions who facilitated connections between the research team and local providers; and (3) interest from both PCP practices and EI programs. The counties vary on sociodemographic factors (see Table 2).
Recruitment and consenting of providers
The research team visited interested PCP practices and EI programs to provide study details and obtain feedback regarding study procedures. Suggestions from providers resulted in adaptations to workshop content, study procedures, survey instruments, and study incentives. Our plan was to partner with 2 PCP practices and 2 EI agencies in each county and enroll a total of 40 PCPs and 80 EI across all 4 counties. PCPs and EI providers were given study consent forms and provided with the opportunity to review them at their leisure, and contact the research team with any questions. PCPs and EI providers who enrolled received compensation for completing surveys at designated intervals ($20/time period).
Recruitment, eligibility, and consenting of families
In both the Pre-SRT and Post-SRT implementation periods, two parent-toddler samples are recruited: an “ASD Concerns” sample and a “No Concerns” sample. The ASD Concerns sample, recruited from both PCP and EI settings, comprises toddlers who either have an ASD diagnosis or meet one or more of the following criteria indicating ASD “risk”: (1) behaviorally-based parental concerns about the presence of ASD; (2) behaviorally-based provider concerns (PCP or EI) about the presence of ASD; (3) having an older sibling with an ASD diagnosis; or (4) parental report of a prior positive screen on a validated ASD screening tool. The planned sample size for the ASD Concerns group was 490 families total (245 Pre-SRT and 245 Post-SRT). The No Concerns sample comprises toddlers whose parents do not endorse any ASD or other concerns; this sample is recruited exclusively from PCP practices to characterize health care experiences of families without concerns about their child’s development. The planned sample size for the ASD Concerns group was 70 families total (35 Pre-SRT and 35 Post-SRT. English- and Spanish-speaking families are recruited from the participating PCP practices and EI programs using materials available in both languages.
For PCP practices, families of 16- to 20-month-olds attending the child’s 18-month well-child visit receive a study flyer and a “permission-to-contact” form by the PCP or medical assistant. For EI programs, recruitment occurs in several ways: flyers are posted in waiting areas, recruitment letters are sent to all parents of toddlers in the program within the desired age range (16–35 months), and EI providers give the study flyer and permission-to-contact form directly to families with ASD concerns. Once parents directly reached out or providers sent the completed permission-to-contact forms, the study personnel conduct a structured telephone screening interview to determine study eligibility and group assignment.
Parents who were interested and eligible to participate were either mailed or emailed copies of the study consent materials. They were provided the opportunity to review the forms at their leisure, and were instructed to call the research team with any questions or concerns. For parents with literacy issues, research staff members guided them through the consent information and provided them the opportunity to verbally ask questions and consent to the study over the phone. Similar to providers, parents who enrolled received $20 compensation for completing surveys at designated intervals.
Measurement and data collection
Self-report survey data are collected from PCPs, EI providers, and parents. REDCap  was used to create a web-based, longitudinal database that is HIPAA compliant and allows for online survey completion. Parent and provider measures are described in Table 3.
Parent measures assess parents’ well-being and health care satisfaction and toddlers’ social communication and receipt of community-based services. All measures are available in English and Spanish and are collected every three months until the toddler reaches 36 months of age. Identical procedures are used for Pre-SRT and Post-SRT cohorts.
PCP and EI provider measures.
Three types of provider measures are used to detect changes in knowledge, beliefs, and practices related to early identification and intervention for toddlers with ASD. The lack of standardized measures of the specific concepts of interest to this project made it necessary to develop our own tools . Provider Practices Surveys contain questions about early ASD screening, referral, treatment, and are completed at five measurement points, twice before introduction of the SRT model (i.e., at the initial data collection point and one-month prior to their training workshop) and three times after its introduction (i.e., 6, 12, and 18 months after SRT training). Provider Checklists identify provider practices employed at the level of individual patients/toddlers. The PCP Checklist is completed immediately after all 18-month well-child visits, regardless of study participation, to gather anonymized information about the presence of ASD risk factors, whether the child was screened, the screening results, and referrals made. The EI Checklist is completed for all children enrolled in the study who attend the EI program, to indicate the type of services provided. Provider Implementation Surveys assess the feasibility and acceptability of using the M-CHAT-R/F, STAT, and RIT, and are completed at 6-, 12-, and 18-months after the SRT training.
Training providers in SRT components.
Participating PCPs, EI providers, and clinic support staff in each county receive training and technical assistance (TA) in implementing the SRT model over a 3-month period (Fig. 2). Training is provided in-person through interactive workshops conducted at each practice site.
Screen and referral training for PCPs.
Training for PCPs comprises two components. The first is a 2-h onsite workshop for each practice. Training emphasizes the rationale and processes for recognizing the early behavioral features of ASD, conducting universal screening at 18 months, using the web-based M-CHAT-R/F, and discussing positive screens and appropriate referrals with parents. Tablets and WiFi “hotspots” are provided to each practice. The second component involves an onsite TA visit to each practice 2 weeks later, to train office and medical staff on using the web-based M-CHAT-R/F system and developing an optimal work flow plan to accommodate the new procedures. Work flow plans delineate: (1) who is responsible for handing parents the tablet containing the web-based M-CHAT-R/F, and collecting it when completed, (2) when and where the tablet will be given to parents (e.g., waiting area vs. exam room), (3) how the M-CHAT-R/F results are displayed on the final screen, and (4) how the results are conveyed to the PCP. The research team remains “on-call” for TA requests and works with office staff to monitor progress and detect barriers to implementation.
Assessment and treatment training for EI providers.
Training for EI providers occurs through two one-day workshops, both involving “hands-on” practice with children. The STAT workshop provides training on the STAT and a developmentally-targeted parent interview to identify ASD risk. Information about discussing ASD with parents is also included. The RIT workshop provides training on using RIT with toddlers at ASD risk, as well as coaching parents in the use of RIT. WiFi hot spots and video recording equipment are provided to each program to enable them to receive performance feedback and online coaching for the STAT and RIT.
Data management, storage, and monitoring
For both the Online M-CHAT-R/F and online versions of the provider and caregiver surveys, the REDCap system was selected as the web application because of its: (1) user friendly interface; (2) capability for programming complex branching logic; (3) security assurances (e.g., HIPAA compliant); (4) cost-effectiveness (i.e., free to partnering universities and institutions); and (5) opportunity for ongoing guidance and support from experts at the UW Institute for Translational Health Sciences. To facilitate data collection, the research team designed and programmed a web-based, longitudinal database to allow PCPs, EI providers, and parents to complete all surveys online; paper-based surveys were also available for request. Data verification for paper surveys was carried out through double data entry and the reconciliation process, which detected random mistakes and also identify systematic deviations from the correct entry of certain fields/items/scores. At the time of enrollment, providers and parents were each assigned a unique numerical code, which they used to complete their surveys and was used to keep their de-identified study data separately from information containing identifying information both in paper and electronic forms. Study data, identified by numerical code only, was shared with the National Database for Autism Research (NDAR) for those parents who gave explicit permission.
There was no independent data monitoring committee or personnel for this RCT as it was considered minimal risk (i.e., participants were completing surveys as primary research activity) and not required by the funding agency. The planned system for handling adverse event involved immediate reporting to the IRB for guidance and management; no adverse events occurred during the Pre-SRT period.
Statistical analyses and power
The overall study design has multiple levels of outcome measurements, collected at multiple time points. Parents and toddlers are nested within different PCP practices or EI programs, providers are nested within programs/practices, and both parents/children and providers are nested within different communities. We will use Generalized Linear Mixed Models (GLMMs) as the primary data analytic strategy to address the study research aims because this model type: (1) accounts for the nested structure, or multiple levels, of the data collected; (2) can be used to model change over time (i.e., both intercept and slope); and (3) can handle missing data for a specific survey without deletion of entire records [37, 38]. Because we expect that multiple providers nested within the same organization may have similar practices (due to clinical practice guidelines in place within different healthcare systems), we will use intraclass correlation coefficients (ICCs) to describe the extent to which providers’ responses are correlated within settings for the outcomes examined at the provider and caregiver/child level. Higher ICCs will reduce our power to detect change over time.
For child outcomes, the planned sample size would allow the detection of small to moderate differences between the Pre- and Post-SRT cohorts at 80% power. For child social communication, for example, the sample size would allow to us to detect a minimum increase of 40% more for the Post-SRT cohort relative to the Pre-SRT cohort assuming a similar amount of variation for both groups. For provider outcomes, based on inclusion of an estimated 40 PCP providers, we would have sufficient sample size to detect approximately 22% point differences between Pre- and Post-SRT intervention reports at 80% power, assuming that 50% of PCPs are implementing universal screening. With a sample of 80 EI providers, we would have sufficient sample size to detect minimum 15% point differences between Pre- and Post-SRT intervention reports at 80% power.
This paper describes a multi-system pragmatic trial of the Screen-Refer-Treat (SRT) service delivery model, a first of its kind, designed to promote an integrated and coordinated approach for early detection and specialized intervention for toddlers suspected of having ASD. Initial process data from the pre-intervention (Pre-SRT) phase highlight some of our successes, challenges, and “lessons learned” with respect to primary study activities involving provider and family recruitment and data collection, which are outlined below.
Recruitment and retention of providers
We were successful in meeting–and exceeding–our enrollment targets for providers (58 PCPs and 87 EI providers; see Table 4). The number of PCP practices per county ranges from 1 to 4, and the number of participating EI program ranges from 1 to 5. However, the process of obtaining provider buy-in and organizational approvals took considerably longer than estimated. During the recruitment phase, the research team made multiple trips (1–3 per site) to meet with the PCP practices and EI agencies; these trips often required up to 2 months advance scheduling. Furthermore, some of the PCP practices required their own approval processes before enrolling providers. The turnaround time for these local approvals required another 2–5 months, significantly extending the start-up phase and delaying the start of data collection by about a year.
Retention of providers has been moderate-to-high (75% for PCPs and 85% for EI providers) over the first 22 months. To promote retention, research team leaders made quarterly visits to each practice and program. This has been challenging for the more distant counties and those with several practices, as scheduling multiple visits on the same day was rarely possible. Other strategies employed to sustain our presence in communities included: having lunch delivered to PCP practices and EI programs; sending providers “SRT Network”-branded posters and mugs; having “check-in” phone calls; and offering incentives for recruitment efforts and for completing surveys. Additionally, for the two counties farthest from Seattle, community liaisons were hired to facilitate ongoing communication between enrolled providers and the research team.
Overall, the choice of a pragmatic trial and stepped-wedge cluster design worked to our advantage for engagement and retention because providers: (1) were active participants in determining procedures for implementing the SRT model, enabling them to optimize efficiency and fit in their sites; and (2) were assured of receiving training and access to new tools. The high rates of retention even in the remote sites likely reflects both our personalized attention and the presence of locally-based liaisons.
Recruitment and retention of pre-SRT families
We experienced significant challenges in contacting families who signed “permission to contact” forms, particularly those recruited from PCP practices, and did not meet our planned enrollment targets for the ASD Concerns group. Of the 894 referrals from PCPs, 260 (29%) enrolled in the study and 243 (27%) completed surveys. 484 (54%) did not respond to our numerous attempts at contact, and of the 410 families we reached via telephone, 22 (5.4%) were ineligible. Recruitment of families from EI programs exhibited the reverse pattern, in that fewer families were referred (n = 151), but a larger proportion of those referred were enrolled (n = 85; 56%) and completed surveys (74; 49%). Among the referred families, 28 (19%) did not respond to our attempts at contact, and 14 of those contacted (11%) were ineligible. Altogether, of the 317 families who enrolled and completed surveys, 65 (21%) were in the ASD Concerns group, 68 (21%) were in the DD Concerns group, and 184 (58%) were in the No Concerns group (see Table 5). Our enrollment of 43 Hispanic families, 14% of all whom enrolled and submitted data, was lower than the 20% we anticipated based on census data.
Some of the challenges we experienced with family enrollment are consistent with reports indicating that families who live outside of metropolitan areas are underrepresented in mental health services research [39, 40]. Known impediments to non-participation, include low population density from which to recruit, lack of familiarity with research as a cultural practice [41, 42], and inadequate research infrastructure to support research participation. Studies that target remote populations might benefit by establishing decentralized research infrastructures to provide a sustained research presence for recruiting and retaining participants [43, 44]. Trusted local staff could function to personally recruit eligible families and consult to PCP and EI staff as they implement intervention and study procedures.
Additional barriers affect research participation for Hispanic families. We found that having Spanish language recruitment materials and surveys was insufficient to meet recruitment goals for this group. Our community liaisons reported that many Hispanic families work as farm laborers, and cited lack of telephone availability during daytime hours and outdated contact information due to frequent moves as potential challenges to research participation. In addition, recent changes in federal immigration policy may have impeded research participation. Future studies may benefit from hiring full-time research team staff who live locally and can dedicate more time building trust and awareness about the research, and potentially conduct in-person follow-up with families regarding study eligibility screening, consenting, and completing surveys.
Additionally, while our original plan was to recruit two groups of families (i.e., those with ASD Concerns and those with No Concerns), it became apparent that both PCPs and EI providers were hesitant to talk to parents about a study that referred to “ASD concerns.” Even when EI providers had ASD concerns about children, they were reluctant to share their concerns with families until it was time for the child to transition out of EI services into school services. Several adaptations were made in an effort to increase recruitment of the ASD Concerns group. First, the criteria for study eligibility were broadened to include a third group of toddlers for whom there were concerns about more general developmental delays. Second, we provided sample scripts and conducted in-person coaching with EI providers about how to introduce the study to parents in an encouraging manner. Third, we extended the recruitment period beyond the initial 6-month window. Although these measures were somewhat effective, our recruitment numbers for the ASD Concerns group remain lower than projected. With only a quarter of sample size as we initially planned, the power to detect our projected effect sizes on many outcomes will be seriously reduced. Still, we plan to carry out all the planned analyses. But the inference will be interpreted with caution. Instead of treating the analysis as confirmatory, we will focus on the direction and magnitude of the effect sizes, which will serve as effect size estimates for future, well powered, confirmatory studies. In addition, the current sample size should still provide enough power (> 90%) to detect an absolute increase of 30% in level-2 screening in EI Pre- vs. Post-SRT periods.
Provider and parent surveys
Of the 58 enrolled PCPs, 44 (76%) completed their first two Practices Surveys. Of the 87 enrolled EI providers, 76 (87%) completed their first two Practices Surveys. Of the 317 parents enrolled who contributed data, 275 (87%) completed surveys at two or more time points.
Patient-level services data
We encountered significant obstacles to the completion of the PCP Checklist and EI Checklists, which were developed to collect information on the services delivered to individual children. While we anticipated that 1120 PCP checklists would be completed by PCPs during the first 6 months of data collection, that milestone was not met until 12 months into the study. Feedback revealed that the PCP Checklist procedure was not well integrated into their office work flow; as a result, strategies for modifying their work flow were developed through consultation with the office manager and PCPs. In addition, checklist completion goals were established for each practice, and incentivized with: (1) their choice of materials for their practice (e.g., books or toys for their waiting area), and (2) performance feedback in the form of run charts, enabling them to compare their checklist completion rates with other (anonymized) practices. We also met with challenges obtaining EI Checklists, as it was difficult for EI programs to identify staff members who had the time and knowledge to provide the requested information. Of the 85 families enrolled in EI, only 54 checklists (64%) were completed. In addition, considerable delays in returning the EI Checklists rendered it difficult to compare parent and provider reports of services delivered.
Provider hesitancy to discuss ASD concerns, time constraints, and challenges in modifying work flow, which are considered significant contributors to the “theory to practice/policy” gap , also emerged as the primary implementation barriers for research activities. Hesitancy to mention ASD concerns was one of the primary reasons that family recruitment rates were lower than expected. Unlike trials that focus on treatment for diagnosed populations/samples, the PCPs and EI providers in the current study had the added challenge of recruiting families whose children did not have an ASD diagnosis, and who may not have been aware of ASD as a potential concern for 18-month-olds. While modifications to recruitment eligibility and materials were made to remove ASD as the focal point, providers still experienced difficulty in presenting recruitment materials to families. Overall, these challenges parallel those documented in other preventive intervention studies [46,47,48], including other ongoing projects within the NIMH ASD PEDS network, which are conducting research on screening tools and intervention for children at risk for ASD . Applying strategies from Quality Improvement  and Implementation Facilitation  approaches (which aim to reduce the “gap” between best practices and community implementation) at the study outset may have prevented some of these implementation barriers from arising.
In sum, the development of the SRT model was inspired by the need for an earlier and more continuous route to specialized services for families of toddlers with ASD or suspected ASD, and we chose a preventive intervention framework to meet that need. We have learned many lessons about study implementation during the pre-intervention phase of trial that may be helpful to others designing future pragmatic trials. For example, allowing for an extended start-up phase of 6–12 months may be needed to obtain provider buy-in, IRB approvals from multiple institutions, input and feedback from providers and key stakeholders, and adjustment of clinic workflow procedures to implement key study procedures. However, this may be challenging within the context of traditional funding mechanisms. In contrast, the alternative of conducting smaller feasibility/pilot studies  is unlikely to reveal the multitude of issues and complexities that arise within a larger scale study, especially one that targets multiple, interacting service delivery systems. Other ASD researchers are urged to consider the use of pragmatic trials to bridge the gap between lab-based advances and community practices. Although the ultimate success of the SRT model is yet to be determined, the results of this study will provide key insights about how to best introduce/disseminate it given that: (1) a network of information and engagement has been established with stakeholders (e.g., Department of Health) and existing infrastructure (i.e., Part C EI); and (2) the model has high ecological validity since it is being evaluated in a “usual care” setting (i.e., community), rather than a carefully controlled clinical lab setting, which likely reduces barriers to future transportability.
Availability of data and materials
The PI and Co-Investigators will have access to the final dataset and a portion of the data will be available on the National Database for Autism Research for researchers that apply through NIMH. Potential authors for future publications will be expected to adhere to American Psychological Association guidelines.
American Academy of Pediatrics
Autism Spectrum Disorder
Continuing education unit
Continuing medical education
Generalized Linear Mixed Models
Health Insurance Portability and Accountability Act
Intraclass correlation coefficients
Individuals with Disabilities Education Act
Institutional Review Board
Modified Checklist for Autism in Toddlers-Revised, with Follow Up
Maintenance of certification
National Database for Autism Research
- NIMH ASD PEDS:
National Institute of Mental Health Autism Spectrum Disorder Pediatric, Early Detection, Engagement and Services
Primary care providers
Randomized controlled trial
Research electronic data capture
Reciprocal Imitation Training
Screening Tool for Autism in Toddlers
University of Washington
- WHO Quality Of Life-BREF:
World Health Organization Quality of Life abbreviated version
Women, Infants, and Children
American Psychiatric Association. Diagnostic and statistical manual of mental disorders (DSM-V). 5th ed. Arlington, VA: American Psychiatric Publishing; 2013.
Baio J, Wiggins L, Christensen DL, et al. Prevalence of autism spectrum disorder among children aged 8 years — autism and developmental disabilities monitoring network. MMWR Surveill Summ. 2018;67:1–23. https://doi.org/10.15585/mmwr.ss6706a1.
Dawson G, Rogers S, Munson J, et al. Randomized, controlled trial of an intervention for toddlers with autism: the early start Denver model. Pediatrics. 2010;125:e17–23. 2009/12/02. https://doi.org/10.1542/peds.2009-0958.
Ingersoll B. Brief report: pilot randomized controlled trial of reciprocal imitation training for teaching elicited and spontaneous imitation to children with autism. J Autism Dev Disord. 2010;40:1154–60.
Kasari C, Gulsrud A, Paparella T, et al. Randomized comparative efficacy study of parent-mediated interventions for toddlers with autism. J Consult Clin Psychol. 2015;83:554–63. https://doi.org/10.1037/a0039080.
Landa RJ, Holman KC, O'Neill AH, et al. Intervention targeting development of socially synchronous engagement in toddlers with autism spectrum disorder: a randomized controlled trial. J Child Psychol Psychiatry. 2011;52:13–21. 2010/12/04. https://doi.org/10.1111/j.1469-7610.2010.02288.x.
Wetherby AM, Guthrie W, Woods J, et al. Parent-implemented social intervention for toddlers with autism: an RCT. Pediatrics. 2014;134:1084–93. https://doi.org/10.1542/peds.2014-0757.
Johnson CP, Myers SM. Identification and evaluation of children with autism spectrum disorders. Pediatrics. 2007;120:1183–215. https://doi.org/10.1542/peds.2007-2361.
Arunyanart W, Fenick A, Ukritchon S, et al. Developmental and autism screening: a survey across six states. Infants Young Child. 2012;25:175–87.
Carbone PS, Behl DD, Azor V, et al. The medical home for children with autism spectrum disorders: parent and pediatrician perspectives. J Autism Dev Disord. 2010;40:317–24. 2009/09/22. https://doi.org/10.1007/s10803-009-0874-5.
Crais ER, Watson LR. Challenges and opportunities in early identification and intervention for children at-risk for autism spectrum disorders. Int J Speech-Lang Pathol. 2014;16:23–9. 2013/12/18. https://doi.org/10.3109/17549507.2013.862860.
Dosreis S, Weiner CL. Autism spectrum disorder screening and management practices among general pediatric providers. J Dev Behav Pediatr. 2006;27:S88–94.
Zuckerman KE, Mattox K, Donelan K, et al. Pediatrician identification of Latino children at risk for autism spectrum disorder. Pediatrics. 2013;132:445–53. https://doi.org/10.1542/peds.2013-0383.
Black JE, Jone TA, Nelson CA, et al. Nearonal plasticity and the developing brain. In: Alessi NE, Coyle JT, Harrison SI, et al., editors. Handbook of child and adolescent psychiatry: basic psychiatric science and treatment. New York: John Wiley & Sons; 1998. p. 31–53.
Fox NA, Calkins SD, Bell MA. Neural plasticity and development in the first two years of life: evidence from cognitive and socioemotional domains of research. Dev Psychopathol. 1994;6:677–96.
Carbone PS, Murphy NA, Norlin C, et al. Parent and pediatrician perspectives regarding the primary care of children with autism spectrum disorders. J Autism Dev Disord. 2013;43:964–72. https://doi.org/10.1007/s10803-012-1640-7.
Mulligan J, MacCulloch R, Good B, et al. Transparency, Hope, and empowerment: a model for partnering with parents of a child with autism Spectrum disorder at diagnosis and beyond. Soc Work Ment Health. 2012;10:311–30. https://doi.org/10.1080/15332985.2012.664487.
Sansosti FJ, Lavik KB, Sansosti JM. Family experiences through the autism diagnostic process. Focus Autism Other Dev Disabil. 2012;27:81–92. https://doi.org/10.1177/1088357612446860.
Tomlin A, Koch SM, Raches C, et al. Autism screening practices among early intervention providers in Indiana. Infant Young Child. 2013;26:74–88. https://doi.org/10.1097/IYC.0b013e31827842b1.
Mrazek PB, Haggerty RJ, Institute of Medicine (U.S.). Committee on Prevention of Mental Disorders., et al. Reducing risks for mental disorders : frontiers for preventive intervention research. Washington, D.C.: National Academy Press; 1994. p. 1. online resource (xxvii, 605 pages)
Ibanez LV, Stone WL, Coonrod EE. Screening for autism in young children. In: Volkmar FR, Paul R, Rogers SJ, et al., editors. Handbook of Autism and Pervasive Developmental Disorders. 4th ed. New Jersey: John Wiley & Sons; 2014.
Robins DL, Casagrande K, Barton M, et al. Validation of the modified Checklist for autism in toddlers, revised with follow-up (M-CHAT-R/F). Pediatrics. 2014;133:37–45. https://doi.org/10.1542/peds.2013-1813.
Chlebowski C, Robins DL, Barton ML, et al. Large-scale use of the modified checklist for autism in low-risk toddlers. Pediatrics. 2013;131:E1121–7. https://doi.org/10.1542/peds.2012-1525.
Barton ML, Dumont-Mathieu T, Fein D. Screening young children for autism spectrum disorders in primary practice. J Autism Dev Disord. 2012;42:1165–74. 2011/08/16. https://doi.org/10.1007/s10803-011-1343-5.
REDCap. Vanderbilt University, 2018.
Harris PA, Taylor R, Thielke R, et al. Research electronic data capture (REDCap)—a metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform. 2009;42:377–81. https://doi.org/10.1016/j.jbi.2008.08.010.
Individuals with disabilities education act 2004.
Stone WL, Coonrod EE, Ousley OY. Brief report: screening tool for autism in two-year-olds (STAT): development and preliminary data. J Autism Dev Disord. 2000;30:607–12. https://doi.org/10.1023/A:1005647629002.
Stone WL, Coonrod EE, Turner LM, et al. Psychometric properties of the STAT for early autism screening. J Autism Dev Disord. 2004;(6):691–701.
Stone WL, McMahon CR, Henderson LM. Screening tool for autism in two-year-olds (STAT) for children under 24 months: an exploratory study. Autism. 2008;12:557–73.
Ingersoll B. Brief report: effect of a focused imitation intervention on social functioning in children with autism. J Autism Dev Disord. 2012;42:1768–73. https://doi.org/10.1007/s10803-011-1423-6.
Ingersoll B, Lalonde K. The impact of object and gesture imitation training on language use in children with autism spectrum disorder. J Speech Lang Hear Res. 2010;53:1040–51. https://doi.org/10.1044/1092-4388(2009/09-0043).
Hussey MA, Hughes JP. Design and analysis of stepped wedge cluster randomized trials. Contemp Clin Trials. 2007;28:182–91. https://doi.org/10.1016/j.cct.2006.05.007.
Patsopoulos NA. A pragmatic view on pragmatic trials. Dialogues Clin Neurosci. 2011;13:217–24.
Loudon K, Treweek S, Sullivan F, et al. The PRECIS-2 tool: designing trials that are fit for purpose. BMJ. 2015;350:h2147. https://doi.org/10.1136/bmj.h2147.
Lewis CC, Weiner BJ, Stanick C, et al. Advancing implementation science through measure development and evaluation: a study protocol. Implement Sci. 2015;10:102. https://doi.org/10.1186/s13012-015-0287-0.
Raudenbush SW and Bryk AS. Hierarchical linear models : applications and data analysis methods. 2nd ed. Thousand Oaks: Sage Publications, 2002, p.xxiv, 485 pages.
Singer JD and Willett JB. Applied longitudinal data analysis : modeling change and event occurrence. Oxford; New York: Oxford University Press, 2003, p.xx, 644 pages.
Baquet CR, Hammond C, Commiskey P, et al. Health disparities research--a model for conducting research on cancer disparities: characterization and reduction. J Assoc Acad Minor Phys. 2002;13:33–40 2002/10/05.
Vander Stoep A, Myers K. Methodology for conducting the Children's attention-deficit hyperactivity disorder Telemental health treatment study in multiple underserved communities. Clin Trials. 2013;10. https://doi.org/10.1177/1740774513494880.
Duran V, Popescu A-D. The challenge of multicultural communication in virtual teams. Procedia Soc Behav Sci. 2014;109:365–9. https://doi.org/10.1016/j.sbspro.2013.12.473.
Tiwari T, Casciello A, Gansky SA, et al. Recruitment for health disparities preventive intervention trials: the early childhood caries collaborating centers. Prev Chronic Dis. 2014;11:E133. https://doi.org/10.5888/pcd11.140140.
Altman DG. Sustaining interventions in community systems: on the relationship between researchers and communities. Health Psychol. 1995;14:526–36.
Arthur MW, Blitz C. Bridging the gap between science and practice in drug abuse prevention through needs assessment and strategic community planning. J Community Psychol. 2000;28:241–55. https://doi.org/10.1002/(SICI)1520-6629(200005)28:3<241::AID-JCOP2>3.0.CO;2-X.
Lau DT, McCaig LF, Hing E. Toward a more complete picture of outpatient, office-based health care in the U.S. Am J Prev Med. 2016;51:403–9. https://doi.org/10.1016/j.amepre.2016.02.028.
Fernandez MA, Eyberg SM. Predicting treatment and follow-up attrition in parent-child interaction therapy. J Abnorm Child Psychol. 2009;37:431–441. 2008/12/20. https://doi.org/10.1007/s10802-008-9281-1.
Mendez JL, Carpenter JL, LaForett DR, et al. Parental engagement and barriers to participation in a community-based preventive intervention. Am J Community Psychol. 2009;44:1–14. https://doi.org/10.1007/s10464-009-9252-x.
Perrino T, Estrada Y, Huang S, et al. Predictors of participation in an eHealth, family-based preventive intervention for Hispanic youth. Prev Sci. 2018;19:630–41. 2016/10/06. https://doi.org/10.1007/s11121-016-0711-y.
Broder-Fingert S, Feinberg E, Silverstein M. Improving screening for autism Spectrum disorder: is it time for something new? Pediatrics. 2018;141. https://doi.org/10.1542/peds.2018-0965.
Batalden PB and Davidoff F. What is “quality improvement” and how can it transform healthcare? BMJ 2007; 16: 2–3. DOI: https://doi.org/10.1136/qshc.2006.022046 % J Quality and Safety in Health Care.
Kirchner MJ, Funes VA, Adzet CB, et al. Quality indicators and specifications for key processes in clinical laboratories: a preliminary experience. Clin Chem Lab Med. 2007;45:672–7. https://doi.org/10.1515/CCLM.2007.122.
Larson EB, Tachibana C, Thompson E, et al. Trials without tribulations: minimizing the burden of pragmatic research on healthcare systems. Healthcare. 2016;4:138–41. https://doi.org/10.1016/j.hjdsi.2015.07.005.
Stone WL, Ibanez LV and Harker C. PCP Practices Survey Unpublished, 2014.
Stone WL and Ibanez LV. PCP Checklist. University of Washington. Unpublished, 2014.
Dingfelder HE, Mandell DS. Bridging the research-to-practice gap in autism intervention: an application of diffusion of innovation theory. J Autism Dev Disord. 2011;41:597–609. 2010/08/19. https://doi.org/10.1007/s10803-010-1081-0.
Stone WL, Ibanez LV and Harker C. EI Provider Practices Survey Unpublished, 2014.
Stone WL and Ibanez LV. EI Checklist. Unpublished, 2014.
Dingfelder HE. The impact of classroom implementation climate on autism intervention fidelity and outcomes. University of Pennsylvania, Philadelphia. Unpublished, 2012.
Stone WL, Ibanez LV and Harker C. Child Health Services Survey Unpublished, 2014.
Stone WL, Ibanez LV and Harker C. Intervention Services Survey Unpublished, 2014.
Abidin RR. Parenting stress index (PSI). Charlottesville, VA: Pediatric Psychology Press; 1990.
Teti DM, Gelfand DM. Behavioral competence among mothers of infants in the first year: the mediational role of maternal self-efficacy. Child Dev. 1991;62:918–29. https://doi.org/10.1111/j.1467-8624.1991.tb01580.x.
Stone WL, Coonrod EE, Pozdol SL, et al. The parent interview for autism-clinical version (PIA-CV) a measure of behavioral change for young children with autism. Autism. 2003;7:9–30.
Bjerre I, Larsson M, Franzon AM, et al. Measure of processes of care (MPOC) applied to measure parent's perception of the habilitation process in Sweden. Child Care Health Dev. 2004;30:123–30.
WHOQOLGroup. Development of the World Health Organization WHOQOL-BREF quality of life assessment. Psychol Med. 1998;28:551–8.
The authors acknowledge the contributions of Allycen Kurup, Roya Baharloo, Juan Pablo Espinosa, Elyanah Posner, and John Hershberger--the lead research assistants--who managed recruitment, enrollment, and data collection at the University of Washington. Additionally, we are thankful for the contributions and guidance from Dana Stevens (Spokane), Cindy Carroll (Yakima), and Jennie Olson (Yakima), who serve as community liaisons for the study. We are extremely grateful for the Primary Care Providers, Early Intervention Providers, and families who give their valuable time to participate in this study.
The current study is funded by the National Institute of Mental Health (R01 MH104302). NIMH did not have input on any aspect of the study design or activities or the decision to submit the report for publication. The Program Officer is Dr. Denise Pintello (email@example.com).
Ethics approval and consent to participate
Ethics approval for this study was granted by the University of Washington Institutional Review Board. Providers and parents who were interested and eligible to participate were either mailed or emailed copies of the study consent materials, and then provided written or typed consent electronically. They were provided the opportunity to review the forms at their leisure, and were instructed to call the research team with any questions or concerns. For parents with literacy issues, research staff members guided them through the consent information and provided them the opportunity to verbally ask questions and consent to the study over the phone.
Consent for publication
WS is co-developer of the Screening Tool for Autism in Toddlers (STAT) and receives royalties from its sales.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Grant support: The current study is funded by the National Institute of Mental Health (R01 MH104302)
About this article
Cite this article
Ibañez, L.V., Stoep, A.V., Myers, K. et al. Promoting early autism detection and intervention in underserved communities: study protocol for a pragmatic trial using a stepped-wedge design. BMC Psychiatry 19, 169 (2019). https://doi.org/10.1186/s12888-019-2150-3
- Autism Spectrum Disorder
- Early detection
- Preventive intervention
- Universal screening
- Stage 2 screening
- Pragmatic trial
- Stepped-wedge design