Interrater reliability (IRR) is a measure of agreement between two or more raters in terms of their ratings of a given sample of items or subjects (Mesmer-Magnus & Waldman, 2005). It is a key component of any research study that involves the use of ratings or rankings, and is a key indicator of the validity of the results. In order to ensure that research studies are valid, it is important to assess the reliability of the raters in order to determine the extent to which the results can be trusted (Mesmer-Magnus & Waldman, 2005).
IRR is typically assessed by having two or more raters rate the same set of items or subjects. This can be done in a number of ways, such as having the raters rate each item or subject separately and then comparing the ratings, or having the raters rate the items or subjects together and then comparing the ratings (Mesmer-Magnus & Waldman, 2005). The most common way to measure IRR is to use a statistical measure called the intraclass correlation coefficient (ICC) (Mesmer-Magnus & Waldman, 2005). The ICC is a measure of the degree of agreement between raters, and is calculated by comparing the average of each rater’s ratings with the average of all of the other raters’ ratings.
IRR is an important concept in social science research, and is used to assess the validity of research findings. For example, in a study of the effects of a particular intervention, if the raters do not have reliable ratings, then it is difficult to draw conclusions from the results of the study. Similarly, in a study of the effectiveness of a particular educational program, if the raters do not have reliable ratings, then the results of the study may be unreliable.
In order to ensure that research studies are valid and reliable, it is important to assess the reliability of the raters. IRR is one of the most important measures of reliability, and is a key component of any research study that involves the use of ratings or rankings.
References
Mesmer-Magnus, J. R., & Waldman, D. A. (2005). Inter-rater reliability: Essential guide to measuring and improving agreement. Thousand Oaks, CA: Sage.