Skip to main content

Corrected analysis of “the effects of bright light treatment on affective symptoms in people with dementia: a 24-week cluster randomized controlled trial” that accounts for clustering and nesting verifies conclusions

Abstract

In this correspondence, we explain the reasoning for invalidity of the analysis choices by Kolberg et al., and provide the results produced using correct statistical procedures for their study design. Reassuringly, we could verify the original conclusions. That is, results of the corrected statistical models are similar to the results of the original analysis. Regardless of the magnitude of difference that corrected statistical methods make, results and conclusions that are derived from invalid methods are unsubstantiated. By verifying the results, we allow the readers to be assured that the published conclusions in the study by Kolberg et al. now rest on a sound evidential basis.

Peer Review reports

Kolberg et al. conducted a cluster-randomized controlled trial (cRCT) to study the effects of Bright Light Treatment (BLT) on behavioral and psychological symptoms of dementia (BPSD) [1] in patients residing in nursing homes. In this cRCT, 8 nursing homes (i.e., clusters) were randomized to either the control or the BLT groups. The published results indicate statistically significant effects of BLT compared to control on mood-related and affective symptoms. Kolberg et al. [1] did not account for clustering effect of the nursing homes (which induces a pattern of correlated data) and nesting (i.e., individuals nested in nursing homes, which limits degrees of freedom for testing the intervention effect) in their analyses.

Clustering and nesting have implications for statistical inference in cRCTs and must be taken into account. In cRCTs, participants within the same cluster potentially have common experiences and characteristics. Thus, “errors” (model residuals) are not independent. Ignoring this potential dependency within clusters (i.e., clustering, most commonly quantified by Intraclass Correlation Coefficient) can result in misestimation of the components of variance [2, 3]. Additionally, degrees of freedom must be adjusted during statistical analysis of cRCTs because “the units of observation (i.e., individual participants) are nested within the units of assignment (i.e., clusters)” [4]. This limits the sample size, and thus, degrees of freedom for accurate estimation of the intervention effect [3]. Results and conclusions derived from statistical tests that ignore clustering and nesting are unreliable.

The authors confirmed our understanding about their statistical analytic choices, and collegially provided access to the deidentified raw data underlying the published results in their paper. In the original study, the R package lme4 was used and random intercepts for individual patients were included in the multilevel linear regression models. This adjusts for correlation due to repeated measurements on each subject, but not for clustering. We re-analyzed the data using the package lme4 multilevel linear regression models including random intercept for individuals nested within nursing homes to account for clustering, and random intercept for individuals to account for the repeated measures. We also adjusted for degrees of freedom to account for the nesting effect of the design.

The p-values of both analyses’ results are shown in Table 1. Reassuringly and similar to the results of the original analysis, results of the corrected statistical models indicate statistically significant time-by-group interaction effects on mood-related and affective symptoms (i.e., BPSD). Thus, through reanalyzing the data using valid methods, we could verify the original conclusions.

Table 1 P-values of time-by-group interaction effects from multilevel linear regression models with and without taking clustering and nesting into account

Regardless of the magnitude of difference that corrected statistical methods make, results and conclusions that are derived from invalid methods are unsubstantiated. That is, unless verified through reanalysis, unsubstantiated results should not be utilized and relied upon by readers. Kolberg et al.’s study addresses a critical need in nursing home settings by providing evidence to support a potentially viable treatment for BPSD among dementia patients. Using rigid statistical approaches is essential for the clinical implications of the study and ensure optimally informed and evidence-based treatment decisions. By our verifying the results, we allow the readers to be assured that the published conclusions in the study by Kolberg et al. now rest on a sound evidential basis.

For upholding the integrity of science, it is vital for reported results and conclusions to be generated using valid statistical procedures for the study design. That is, the validity of statistical methods is strictly a function of pre-specifiable required conditions derived from theory, and these required conditions do not vary with empirical context. In contrast, the robustness of statistical methods can depend on empirical factors [5], which may vary by field. Scholars in and outside of aging, dementia, and sleep research are advised to incorporate and report clustering and nesting adjustments in their analyses of cRCTs. When this is not done, it is advisable to share data for others to apply and report on corrected statistical analyses of these data to verify or correct conclusions.

Availability of data and material

Statistical code is available at https://osf.io/avb2k/.

Dr. Flo-Groeneboom (the senior author of https://doi.org/10.1186/s12888-021-03376-y) provided us with temporary remote access to the dataset used for this re-analysis. We did not have any special access privileges that others would not have.

Abbreviations

BLT:

Bright Light Treatment

cRCT:

cluster Randomized Controlled Trial

BPSD:

Behavioral and Psychological Symptoms of Dementia

References

  1. Kolberg E, Hjetland GJ, Thun E, Pallesen S, Nordhus IH, Husebo BS, et al. The effects of bright light treatment on affective symptoms in people with dementia: a 24-week cluster randomized controlled trial. BMC Psychiatry. 2021;21(1):377. https://doi.org/10.1186/s12888-021-03376-y.

    Article  PubMed  PubMed Central  Google Scholar 

  2. Brown AW, Li P, Bohan Brown MM, Kaiser KA, Keith SW, Oakes JM, et al. Best (but oft-forgotten) practices: designing, analyzing, and reporting cluster randomized controlled trials. Am J Clin Nutr. 2015;102(2):241–8. https://doi.org/10.3945/ajcn.114.105072.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  3. Murray DM, Varnell SP, Blitstein JL. Design and Analysis of Group-Randomized Trials: A Review of Recent Methodological Developments. Am J Public Health. 2004;94(3):423–32. https://doi.org/10.2105/AJPH.94.3.423.

    Article  PubMed  PubMed Central  Google Scholar 

  4. National Institutes of Health. Research Methods Resources: Group- or Cluster-Randomized Trials (GRTs). Available from: https://researchmethodsresources.nih.gov/methods/grt.

  5. Rasch D, Guiard V. (2004). The robustness of parametric statistical methods. Psychology Science, 46(2), 175–208.Available from: https://psycnet.apa.org/record/2004-19989-001.

Download references

Acknowledgements

Not applicable

Funding

Authors supported in part by NIH grants R25DK099080, R25HL124208, U24AG056053, P30AG050886, and the Gordon and Betty Moore Foundation. The opinions expressed are those of the authors and do not necessarily represent those of the NIH or any other organization.

Author information

Authors and Affiliations

Authors

Contributions

Conceptualization: DBA; Analyses: YJ-N and LGA; Writing original draft: YJ-N; Review and editing: DBA, LGA, ABS, CJV, MJR, RJH. All authors approved the final draft.

Corresponding author

Correspondence to David B. Allison.

Ethics declarations

Ethics approval and consent to participate

Not applicable

Consent for publication

Not applicable

Competing interests

Dr. Allison and his institution (Indiana University) have received payments for consultation, grants, contracts, in-kind donations, and contributions from multiple for-profit and not-for-profit entities interested in statistical design and analysis of experiments, but not directly related to the research questions addressed in the paper question.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Jamshidi-Naeini, Y., Golzarri-Arroyo, L., Siddique, A.B. et al. Corrected analysis of “the effects of bright light treatment on affective symptoms in people with dementia: a 24-week cluster randomized controlled trial” that accounts for clustering and nesting verifies conclusions. BMC Psychiatry 23, 683 (2023). https://doi.org/10.1186/s12888-023-05180-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s12888-023-05180-2