Ranking Groups' Abilities - Is It Always Reliable? Academic Article uri icon

abstract

  • The huge technological improvement in data processing and the globalization have increased the demand for and the supply of indices that quantify the consequences of a policy. However, there are certain cases that quantification may be misleading in the sense that it gives the impression of an accurate measurement while in reality it is not. This paper presents such an example in the area of education. Ability is a latent variable. As such, it has to be measured indirectly, using questions of varying levels of difficulties. The larger the number of questions a student answers correctly, the higher the grade and the higher the estimated ability. Groups (i.e. classes, schools, ethnic groups, etc) are ranked in a similar way, based on the average grades of their students. This paper argues that the inability to directly measure ability may cause the ranking of groups to be meaningless, in the sense that one can compose an alternative legitimate exam that will reverse the ranking of groups' average abilities. This case may occur, although the ranking of individuals according to ability may be correct. This problem percolates into other cases of aggregation of grades, such as the use of grades in a regression. The paper provides an easy way to check whether the condition for reversing the order of the ranks is possible. In addition, we report on an empirical study, based on grades achieved in exams in Israel, where in approximately 40 percent of the cases tested, it is possible to change the ranking of the groups by using an alternative exam.

publication date

  • January 1, 2009