Hostname: page-component-6b989bf9dc-wj8jn Total loading time: 0 Render date: 2024-04-14T16:59:14.824Z Has data issue: false hasContentIssue false
  • English
  • Français

Not Seeing Clearly With Cleary: What Test Bias Analyses Do and Do Not Tell Us

Published online by Cambridge University Press:  07 January 2015

Adam W. Meade  and
Scott Tonidandel
Adam W. Meade*
Affiliation:
North Carolina State University
Scott Tonidandel
Affiliation:
Davidson College
*
E-mail: awmeade@ncsu.edu, Address: Department of Psychology, North Carolina State University, Campus Box 7650, Raleigh, NC 27695-7650
Get access Rights & Permissions [Opens in a new window]

Abstract

In recent decades, the Cleary (1968) approach for testing for differences in regression lines among demographic groups has been codified as a central approach to evaluate a test for bias. However, this approach is fraught with numerous shortcomings, a preponderance of implicit assumptions, and outcomes that are not sufficient to conclude that there is a problem with a test. We believe these shortcomings are poorly understood by many industrial–organizational (I–O) psychologists, that this method for evaluating test bias is overrelied on by our profession, and that it is interpreted improperly by those wishing to evaluate tests for bias in applied settings. Moreover, eliminating differential prediction may be impossible in some cases, undesirable in others, and places an undue burden on organizational researchers.

Type
Focal Article
Information
Industrial and Organizational Psychology , Volume 3 , Issue 2 , June 2010 , pp. 192 - 205
Copyright
Copyright © Society for Industrial and Organizational Psychology 2010 

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Footnotes

*

Department of Psychology, North Carolina State University

**

Department of Psychology, Davidson College.

References

Aguinis, H., Boik, R. J., & Pierce, C. A. (2001). A generalized solution for approximating the power to detect effects of categorical moderator variables using multiple regression. Organizational Research Methods, 4, 291–323.Google Scholar
Aguinis, H., Culpepper, S. A., & Pierce, C. A. (in press). Revival of test bias research in preemployment testing. Journal of Applied Psychology.Google Scholar
Aguinis, H., Culpepper, S. A., & Pierce, C. A. (2009). Revival of test bias research in preemployment testing . Manuscript submitted for publication.Google Scholar
Aguinis, H., & Smith, M. A. (2007). Understanding the impact of test validity and bias on selection errors and adverse impact in human resource selection. Personnel Psychology, 60, 165–199.Google Scholar
Aguinis, H., & Stone-Romero, E. F. (1997). Methodological artifacts in moderated multiple regression and their effects on statistical power. Journal of Applied Psychology, 82, 192–206.Google Scholar
Allen v. Alabama State Board of Education. U.S. Dist. LEXIS 123 (M.D. Al 2000).Google Scholar
American Educational Research Association, American Psychological Association, & National Council on Measurement in Education (AERA/APA/NCME). (1999). Standards for educational and psychological testing. Washington, DC: American Educational Research Association.Google Scholar
Bartlett, C. J., & O’Leary, B. (1969). A differential prediction model to moderate the effects of heterogeneous groups in personnel selection and classification. Personnel Psychology, 22, 1–19.Google Scholar
Bauer, T. N., Maertz, C. P. J., Dolen, M. R., & Campion, M. A. (1998). Longitudinal assessment of applicant reactions to employment testing and test outcome feedback. Journal of Applied Psychology, 83, 892–903.Google Scholar
Camilli, G., & Shepard, L. A. (1994). Methods for identifying biased test items. Thousand Oaks, CA: Sage.Google Scholar
Campbell, J. P. (1996). Group differences and personnel decisions: Validity, fairness, and affirmative action. Journal of Vocational Behavior, 49, 122–158.Google Scholar
Cleary, T. A. (1968). Test bias: Prediction of grades of Negro and White students in integrated colleges. Journal of Educational Measurement, 5, 115–124.Google Scholar
Cole, N. S. (1973). Bias in selection. Journal of Educational Measurement, 10, 237–255.Google Scholar
Connecticut v. Teal, 457 U.S. 440 (1982).Google Scholar
Cormier v. P.P.G. Industries, Inc. U.S. Dist. LEXIS 13634. (W.D. La 1981).Google Scholar
Darlington, R. B. (1971). Another look at “cultural fairness.” Journal of Educational Measurement, 8, 71–82.Google Scholar
Guion, R. M. (1998). Assessment, measurement, and prediction for personnel decisions. Mahwah, NJ: Lawrence Erlbaum.Google Scholar
Hartigan, J. A. E., & Wigdor, A. K. E. (1989). Fairness in employment testing: Validity generalization, minority issues, and the General Aptitude Test Battery. Washington, DC: National Academy Press.Google Scholar
Hough, L. M., Oswald, F. L., & Ployhart, R. E. (2001). Determinants, detection and amelioration of adverse impact in personnel selection procedures: Issues, evidence and lessons learned. International Journal of Selection and Assessment, 9, 152–194.Google Scholar
Hulin, C. L., Drasgow, F., & Parsons, C. (1983). Item response theory: Application to psychological measurement. Homewood, IL: Dow Jones-Irwin.Google Scholar
Hunter, J. E., & Schmidt, F. L. (1976). Critical analysis of the statistical and ethical implications of various definitions of test bias. Psychological Bulletin, 83, 1053–1071.Google Scholar
James, L. R. (1980). The unmeasured variables problem in path analysis. Journal of Applied Psychology, 65, 415–421.Google Scholar
Jeanneret, R. (2005). Professional and technical authorities and guidelines. In Landy, F. J. (Ed.), Employment discrimination litigation: Behavioral, quantitative, and legal perspectives. San Francisco: Jossey-Bass.Google Scholar
Jensen, A. (1980). Bias in mental testing. New York: Free Press.Google Scholar
Linn, R. L., & Werts, C. E. (1971). Considerations for studies of test bias. Journal of Educational Measurement, 8, 1–4.Google Scholar
Martocchio, J. J., & Whitener, E. M. (1992). Fairness in personnel selection: A meta-analysis and policy implications. Human Relations, 45, 489–506.Google Scholar
Mauro, R. (1990). Understanding L.O.V.E. (left out variables error): A method for estimating the effects of omitted variables. Psychological Bulletin, 108, 314–329.Google Scholar
Maxwell, S. E., & Arvey, R. D. (1993). The search for predictors with high validity and low adverse impact: Compatible or incompatible goals? Journal of Applied Psychology, 78, 433–437.Google Scholar
Meade, A. W., & Fetzer, M. (2009). Test bias, differential prediction, and a revised approach for determining the suitability of a predictor in a selection context. Organizational Research Methods, 12, 738–761.Google Scholar
McNemar, Q. (1975). On so-called test bias. American Psychologist, 30, 848–851.Google Scholar
Petersen, N. S., & Novick, M. R. (1976). An evaluation of some models for culture-fair selection. Journal of Educational Measurement, 13, 3–29.Google Scholar
Roth, P. L., Bobko, P., & Switzer, F. S. I. (2006). Modeling the behavior of the 4/5ths rule for determining adverse impact: Reasons for caution. Journal of Applied Psychology, 91, 507–522.Google Scholar
Sackett, P. R., Laczo, R. M., & Lippe, Z. P. (2003). Differential prediction and the use of multiple predictors: The omitted variables problem. Journal of Applied Psychology, 88, 1046–1056.Google Scholar
Sackett, P. R., & Wilk, S. L. (1994). Within-group norming and other forms of score adjustment in preemployment testing. American Psychologist, 49, 929–954.Google Scholar
Schmidt, F. L., Pearlman, K., & Hunter, J. E. (1980). The validity and fairness of employment and educational tests for Hispanic Americans: A review and analysis. Personnel Psychology, 33, 705–724.Google Scholar
Society for Industrial and Organizational Psychology (SIOP) (2003). Principles for the validation and use of personnel selection procedures. Bowling Green, OH: Society for Industrial and Organizational Psychology.Google Scholar
Terris, W. (1997). The traditional regression model for measuring test bias is incorrect and biased against minorities. Journal of Business and Psychology, 12, 25–37.Google Scholar
U.S. Equal Employment Opportunity Commission, Civil Service Commission, Department of Labor, and Department of Justice. (1978). Uniform guidelines on employee selection procedures. Federal Register, 43, 38290–38315.Google Scholar