Skip to main content

Advertisement

Springer Nature is making SARS-CoV-2 and COVID-19 research free. View research | View latest news | Sign up for updates

Table 5 Interaction of various sources of variance on reliability

From: The reproducibility of psychiatric evaluations of work disability: two reliability and agreement studies

Illustration of the interaction of various sources of variance and their impact on the reliability measure ICC.
General formula for ICCabs.agree [25]: \( \frac{\sigma_{Patients}^2}{\sigma_{Patients}^2+{\sigma}_{Psychiatrists}^2+{\sigma}_{Residuals}^2\ } \)
Example 1 - Analogy to the situation observed in RELY 1: the ICC is calculated based on a patient variance of 500, a psychiatrist variance of 100 and a large residual (unexplained) variance of 500.
ICC = \( \frac{\mathbf{500}}{\mathbf{500}+100+500} = 0.45 \) which corresponds to a fair discrimination of patients [26]
Example 2 - Analogy to the situation observed in RELY 2: The ICC is calculated with a patient variance of 250, a psychiatrist variance of 50 and a large residual (unexplained) variance of 250.
ICC = \( \frac{\mathbf{250}}{\mathbf{250}+50+250} = 0.45 \) which corresponds to a fair discrimination of patients (equal to example 1)
Despite reduction of total variance, the proportionate reduction of variance across all sources of variance results in an ICC of 0.45 identical to example 1. Despite reduction of variance by half, the ability to discriminate patients in their ability to work did not change.
Example 3 - Typical situation for a reliable instrument: Most variance is explained by patient variance, with little psychiatrist variance and residual variance: patient variance of 500, psychiatrist variance of 25, and residual variance of 75. As a result, expert variance and residual variance contribute little to the total variance, indicating low measurement error. This allows excellent discrimination among patients. ICC = \( \frac{\mathbf{500}}{\mathbf{500}+25+75} = 0.83 \)