Why Interrater Reliability Matters

As an observation-based tool, AEPS®-3 provides an authentic assessment of a child’s skills and abilities and generates useful information for instruction and intervention. However, one challenge in using observation-based measures is the need to trust the judgements of professionals conducting the assessments. That’s why monitoring interrater reliability is so important.

What is interrater reliability?

Interrater reliability (IRR) refers to whether different people administering the same assessment tool can do so in a consistent way. Because a child’s behavior can vary from one day to the next, interrater reliability is determined by having two or more individuals observe the child at the exact same time, using the same metric. Once the observation is complete, the ratings of the observers are compared. The more similar the ratings, the higher the reliability is between the observers.

Why check interrater reliability?

An assessment can’t be considered authentic and accurate if it can’t be conducted and scored as intended. The purpose of checking interrater reliability is to ensure that the assessment score is an accurate representation of the child. Higher interrater reliability indicates that the assessment is a more reliable gauge of the child’s behavior. Checking IRR, then, contributes to more meaningful assessments for young children.

How does the AEPS-3 IRR Certification help?

Now updated for AEPS-3, this field-tested online certification module helps your program ensure that AEPS-3 is used accurately to promote real progress for every child. The module includes a narrated training presentation with video examples, so that users can practice and prepare for the certification test. Users who complete the test and achieve 80% overall agreement with the gold standard scoring receive an AEPS Interrater Reliability Certification that lasts for three years. When AEPS-3 users are certified, you can be confident that your assessment information is reliable: for every child, across observers, across different circumstances, and on different days.

Learn more about the AEPS-3 IRR certification and order it for your program here.

This post includes information adapted from Assessing Young Children in Inclusive Settings, Second Edition, by Kristie Pretti-Frontczak, Ph.D., & Jennifer Grisham, Ed.D., with Lynn D. Sullivan, M.Ed.