The Intraclass Correlation Coefficient (ICC) and the Z Fisher Transformation for the 95% Confidence Interval

Case Scenario

Suppose that there are 10 subjects with eyelid measurements from three different observes (making the total sample = 30), who used a new computerized technique to get eyelid images and later measured the parameters using image J software. What would you do to test for reliability of measurements and how would you go by performing this analysis? Please note that the dependent variable (eyelid measurement) is a continuous variable. This is a real example.

Approach

The best measure of reliability for continuous data is the Intraclass Correlation Coefficient (ICC). The ICC would inform you about the inter-rater reliability, where the higher the ICC the less unique information each additional measure provides. 

Where S²w is the pooled variance within subjects, and S²b is the variance of the measurements between subjects. S²b + S²w is the total variance. As such, the ICC is interpreted as the proportion of total variance accounted for by the within-subject variation.

You would usually want to calculate a 95% confidence interval along with the ICC. However, because the ICC is a bounded parameter (is only between 0 and 1), it is better to transform the ICC to an unbounded scale and do the confidence interval calculation on that scale (and then go back to the original scale once you get the limits). You would need to calculate the 95% CI of the Fisher's Z transformation of the ICC and then convert these limits to the ICC scale. 

The SAS code below is an example of how you can do this using NLMIXED for a random intercept model:

/* random intercept model for Palpebral */

proc nlmixed data=eyelid.interod  gconv=1e-12;

parms b0=3 varu=1 vare=1;

mean = b0 +u;

model Palpebral ~ normal(mean,vare);

random u ~ normal(0,varu) subject=id;

estimate 'icc' varu / (varu + vare);

estimate 'Fisher z of icc' 0.5 * log((1 + (varu / (varu + vare))) / (1 - (varu / (varu + vare))));

run;

This code will give you an output that includes the below:

Additional Estimates

                                             Standard

Label                   Estimate      Error     DF   t Value   Pr > |t|    Alpha      Lower      Upper

icc                       0.9740     0.01399      9     69.64     <.0001      0.05       0.9424     1.0057

Fisher z of icc      2.1651       0.2727      9      7.94      <.0001      0.05       1.5483     2.7819

You can get the limits on the icc scale using 0.9740 +/- 1.96*0.01399 (which presumably is what is shown in the output as lower and upper limits; values may slightly differ when calculating by hand versus the automated).

Again, the Fisher's Z limits are (1.5483, 2.7819) which also presumably is from:

2.1651 +/- 1.96*0.2727

You would then use the conversion formula to get the 95% CI on the ICC scale

r = (exp(2z) - 1) / (exp(2z) + 1)

In this example you will get the 95% CI as (0.914-0.992). This seems like to be a much better fit. There are times when it may not matter whether you get the 95% CI with the raw scale or the Fisher-transformed scale. However, in certain cases like this (with a high ICC), you definitely need the Fischer z-transformation.

----------------------------------------------------------------------------------------------------------

References and Additional Readings:

Euser AM, Cessie S, Finken MJJ, Wit JM, Dekker FW. Reliability studies can be designed more efficiently by using variance components estimates from different sources. J Clin Epidemiol 2007; 60: 1010-1014.

Longitudinal course at the SPH, UIC given by Dr. Don Hedeker http://tigger.uic.edu/~hedeker/ml.html