A Comparison of the Standardization and IRT Methods of Adjusting Pretest Item Statistics Using Realistic Data

Shun-Wen Chang, National Taiwan Normal University
Bradley A. Hanson, ACT, Inc.
Deborah J. Harris, ACT, Inc.

Paper presented at the Annual Meeting of the American Educational Research Association (Seattle, April, 2001)

Abstract: The requirement of large sample sizes for calibrating items based on IRT models is not easily met in many practical pretesting situations. Although classical item statistics could be estimated with much smaller samples, the values may not be comparable across different groups of examinees. This study extended Chang, Hanson, and Harris (2000) by further exploring the standardization method and comparing its effectiveness with the one-parameter (1PL) and three-parameter (3PL) logistic IRT models in adjusting pretest item statistics with small sample sizes, using more realistic data than the previous study.

Based on the realistic data generated from a 50-dimensional MIRT model, the 3PL model performed better than the 1PL or standardization method in recovering both the population p-values and point biserial correlations. The standardization method outperformed the 1PL model in recovering the population point biserial correlations, but not in recovering the population p-values. The performance of the methods was also evaluated using the real pretest data of a high-stakes test. In terms of recovering the p-values and point biserial correlations for the real data, the 1PL model produced the most satisfactory results. The 3PL model performed worst in terms of recovering the p-values for the real data, and the standardization methodperformed worst in recovering the point biserial correlations for the real data.

Due to the very limited number of conditions studied, one must be cautious about making conclusions about the standardization method relative to IRT methods based on these studies. The standardization method appears to be a viable alternative to IRT methods that may be simpler to implement, although these results do not suggest that it will produce more accurate results.

Download paper in PDF format (110 KB). Version 4.0 or later of Adobe Acrobat Reader (which is freely available) is needed to view this paper.

Return to Brad Hanson's Home Page

Return to OpenIRT Home Page

Last updated: November 16, 2014.