Rethinking Reliability for Innovative Assessments in Mathematics and Science
Goal:This project addresses both methodological and substantive issues in the measurement of reliability of innovative assessments in science and mathematics.

Description:
Reliability analysis has become routine with little consideration of the assumptions regarding learning and performance associated with their use. While it is acknowledged that alternative assessments involve a greater level of cognitive complexity than traditional tests, current approaches towards reliability analysis incorporate a trait-based theory of learning and performance in the test domain.

The failure to recognize and acknowledge the theory of learning and performance underlying a reliability analysis can lead to an incomplete picture of the precision of an assessment procedure and can therefore constrain the kinds of assessment practices accepted by the educational community.

Methodologically, this project develops a new approach to estimating reliability that accurately reflects the precision of an assessment procedure involving cognitively complex tasks and culturally diverse test-takers. This approach incorporates structural equation models which reflect the diverse ways people think and learn.

Structural equation models are developed to compute and construct relevant and irrelevant variance components and are applied to estimating the reliability of innovative assessments under three different approaches: theory fit, facet generalization, and prophecy analysis. The research described above adapts reliability theory to the cognitive complexity of innovative assessments in mathematics and science and maintains the vigilance against sloppy assessment.

Substantively, the project produces estimates of the reliability of performance assessment which acknowledge the construct relevance of cognitively complex tasks and culturally diverse test takers. To address the reliability of performance-based assessments, the cooperating high school is working with investigators to train teachers and administrators in developing, administrating, and scoring performance-based classroom assessments. The expected outcomes are:

  • A set of performance assessments with well understood sources of difficulty.
  • Evidence concerning the reliability of cognitively complex performance assessments conditional on group and instructional variables.
Funding:
National Science Foundation

Directors:
Philip Smith and
Anne Teeter

Project Period:
January 1998 - August 2001
 
Retargeting Pixel