Many investment firms and services use the “information coefficient”—the correlation between predicted and actual returns—to assess the performance of analysts or valuation models. What many may fail to realize is that any IC based on fewer than several thousand stocks will suffer from significant sampling error.

For example, an IC based on predictions for all the stocks on the New York Stock Exchange is unlikely to be higher than 0.10. But an IC based on 30 of those stocks will vary randomly by plus or minus 0.35! Thus sampling error in the 30-stock IC is likely to create an illusion of massive variation over time, or over analysts or over investment models.

Analysis of IC data on the Wells Fargo and Value Line valuation models reveals that the large differences in the models’ ICs over time are due solely to sampling error. Analysis of analysts’ ICs reveals, similarly, that the differences between analysts’ ICs can be attributed entirely to sampling error. If there are differences in analysts’ performances, they must be measured in some manner other than comparing ICs.