Educational Achievement Standards

NAGB's Approach Yields Misleading Interpretations Gao ID: PEMD-93-12 June 23, 1993

GAO reviewed the approach used by the National Assessment Governing Board (NAGB) to establish standards for student performance in mathematics. NAGB reported that few American students had reached these standards. Expert reviewers noted several technical problems with the approach, which had several novel features, and questioned the results. This report extensively analyzes the validity of issues raised by NAGB's approach, examines alternative approaches to setting standards for performance, and reviews NAGB's capacity to provide sound guidance on technical issues. GAO found that NAGB's 1990 standard-setting approach was procedurally flawed and that the interpretations NAGB gave to the resulting scores were of doubtful validity. GAO recommends that the question of how to set and interpret performance standards be reopened and that structures and procedures for governing the assessment be reviewed to ensure that policies are technically sound as well as responsive to constituent interests.

GAO found that: (1) NAGB based its approach on a well-known standard-setting method but modified the method in untested ways; (2) the 1990 standard-setting approach NAGB used was unusual because achievement levels were to reflect mastery of different material, not differences in overall performance, panelists did not use consensus-based standards, and panelists were not assisted in judging students that met expectations for lower levels; (3) the NAEP scale can be used to express standards for overall performance on grade-level materials; (4) because of the way NAEP is designed, the current NAEP scale is not a good way to measure students' knowledge in specific areas; (5) if the measurement of knowledge of course content is desired, new tests will need to be developed; (6) NAGB designed and implemented its achievement levels measurement approach without adequate technical information; and (7) NAGB knowledge, resources, and procedures do not provide reasonable assurance that work done at its direction will be technically sound.

Recommendations

Our recommendations from this work are listed below with a Contact for more information. Status will change from "In process" to "Open," "Closed - implemented," or "Closed - not implemented" based on our follow up work.

Director: Team: Phone:


The Justia Government Accountability Office site republishes public reports retrieved from the U.S. GAO These reports should not be considered official, and do not necessarily reflect the views of Justia.