>>Management Pages

Reliability Page

This page lets you test the inter-rater reliability of your rubrics and establish how reliable they are across a group of assessments. You can comparing rubric reliability by asking other randomly chosen assessors to reassess work and comparing the new scores with the old.

The rubric reliability process will try to find a certain number of assessments for every rubric, and then send them to assessors to be scored again. Visible only to Administrators, the results are presented on the Reliability Page, and the settings can be adjusted by clicking the Allocate Re-assessments link.

Overall Reliability

This section, at the top of the Reliability Page, gives you an overview of the reliability assessment process to date (see below). You are shown how many re-assessments have been allocated, how many of those have been completed, and statistics for the comparisons so far. The results shown here will give the best indication of rubric reliability if multiple reassessments have been spread evenly across rubrics. The more widely and equally you gather your data, the more accurate the results.

The statistics can be interpreted based upon the following guidelines:

• Mean - A positive number here indicates that the original assessments were higher, on average, than the re-assessments (i.e. the original assessments were optimistic). For example, a score of 1.0 tells you that the original assessments are, on average, 1 point higher than the re-assessments that have just been made. A score of 0 indicates the assessments are the same, and a negative score indicates a lower original grading of the work.
  
  
• Std Deviation -This statistical measure indicates how spread out numbers are from the average value, ie. the amount of random variation between assessments and re-assessments (in score units). As a general guide, 0.5 is good and there should be some cause for concern if the value is greater than 1.
  
  

Reliability on Specific Rubrics

The rubrics listed under Reliability on Specific Rubrics give you a more detailed breakdown of rubric reliability throughout your institution (see below). For each rubric you can see how many re-assessments have been allocated, how many of those have been completed, and statistics for the comparisons so far.

>>Management Pages

What do you want to do on this page?

Work with re-assessments

• Allocate re-assessments - Decide which rubrics will be examined, and who will re-assess them.