Interrater Agreement Ordinal Data

Interrater Agreement Ordinal Data

Summary of ICC statistical parameters for ordination, interval or ratio variables. The CCI evaluation (McGraw- Wong, 1996) was conducted using an inter-mediated CCI to assess the extent to which coders provided consistency in their sensitivity beyond the subjects. The resulting CCI was in the excellent ICC range of 0.96 (Cicchetti, 1994), indicating that the coders had a high degree of convergence and indicate that empathy was assessed similarly in donors. The high CCI suggests that independent coders have introduced a minimum amount of measurement errors and that, therefore, statistical performance for subsequent analyses is not significantly reduced. Empathy assessments were therefore deemed appropriate to be used in the hypothesis testing of the hypothesis in this study. For completely cross-referenced designs with three or more coders, Light (1971) proposes to calculate Kappa for all pairs of coders, and then use the arithmetic average of these estimates to provide an overall index of match. Davies and Fleiss (1982) propose a similar solution using the average P (e) between all pairs of coders to calculate a cappa-type statistic for multiple coders. Light, Davies and Fleiss solutions are not available in most statistical packages; However, the light solution can be easily implemented by calculating Kappa for all pairs of coders using statistical software, and then manually calculating the arithmetic average. The spSS and the R-pack require users to indicate a single or two-way model, an absolute type of match or consistency, as well as individual or medium units. The design of the hypothetical study provides information on the correct choice of ICC variants. Note that the SPSS, but not the R-irr package, allows a user to indicate random or mixed effects, the calculation and results are identical for random and mixed effects. For this hypothetical study, all subjects were evaluated by all coders, meaning that the researcher should probably use a two-way ICC model, because the design is completely cross-referenced and an average CCI unit of measurement, because the researcher is probably interested in the reliability of the average evaluations provided by all coders. The researcher is interested in assessing the degree of correspondence between the coder`s assessments, so that higher ratings of one coder corresponded to higher ratings of another coder, but not to the extent that the coders agreed on the absolute values of their ratings, which justifies a type of ICC consistency.

The coders were not randomly selected, so the researcher is interested in knowing how much the coders agreed on their assessments in the current study, but not to generalize these assessments to a larger population of coders, which justifies a mixed model. The data presented in Table 5 are in their final form and are not further processed, so these are the variables on which an analysis of the IRR should be performed. Kappa statistics are used to assess the agreement between two or more advisors if the scale of measurement is categorical.


(转载本站文章请注明作者和出处 酷 壳 – CoolShell ,请勿用于任何商业用途)

——=== 访问 酷壳404页面 寻找遗失儿童。 ===——
好烂啊有点差凑合看看还不错很精彩 (没人打分)