Table 5 Interaction of various sources of variance on reliability

Illustration of the interaction of various sources of variance and their impact on the reliability measure ICC.

General formula for ICC_abs.agree [25]: \( \frac{\sigma_{Patients}^2}{\sigma_{Patients}^2+{\sigma}_{Psychiatrists}^2+{\sigma}_{Residuals}^2\ } \)

Example 1 - Analogy to the situation observed in RELY 1: the ICC is calculated based on a patient variance of 500, a psychiatrist variance of 100 and a large residual (unexplained) variance of 500.

ICC = \( \frac{\mathbf{500}}{\mathbf{500}+100+500} = 0.45 \) which corresponds to a fair discrimination of patients [26]

Example 2 - Analogy to the situation observed in RELY 2: The ICC is calculated with a patient variance of 250, a psychiatrist variance of 50 and a large residual (unexplained) variance of 250.

ICC = \( \frac{\mathbf{250}}{\mathbf{250}+50+250} = 0.45 \) which corresponds to a fair discrimination of patients (equal to example 1)

Despite reduction of total variance, the proportionate reduction of variance across all sources of variance results in an ICC of 0.45 identical to example 1. Despite reduction of variance by half, the ability to discriminate patients in their ability to work did not change.

Example 3 - Typical situation for a reliable instrument: Most variance is explained by patient variance, with little psychiatrist variance and residual variance: patient variance of 500, psychiatrist variance of 25, and residual variance of 75. As a result, expert variance and residual variance contribute little to the total variance, indicating low measurement error. This allows excellent discrimination among patients. ICC = \( \frac{\mathbf{500}}{\mathbf{500}+25+75} = 0.83 \)