GAO Evaluation of Report by Technassociates, Inc., on the Evaluation of Written Products Delivered by the National Center for Research in Vocational Education

In response to a congressional request, GAO reviewed a contracted evaluation of the National Center for Research in Vocational Education to determine whether the methodology used in producing the report and the support for its findings and recommendations were sound.

GAO found that the methodology used in the study was a reasonable way of evaluating some aspects of the quality of the Center's work. However, the report failed to state the size of the population from which the samples of written products were drawn or maintain the distinction between stratified random and judgmental samples of Center documents. In addition, GAO found: (1) an absence of any reporting of the number of responses on which reported results are based; and (2) the absence of a description of the method by which a consensus rating was achieved or even whether such a rating was achieved when a document was reviewed by more than one panel member. Therefore, it was difficult to support the findings. GAO found support for the findings that: (1) the Center had complied with federal contractual requirements but did not meet the larger purposes of legislative intent; (2) the Center's products added little to the existing knowledge about vocational education; and (3) the Center's written products provided little basis in research for improvement in the effectiveness or equity of vocational education services. GAO could not determine the adequacy of support for findings concerning the quality of the Center's products and of its management. GAO found that recommendations concerning the Center's need: (1) to narrow its agenda; (2) build an adequate information base; and (3) for greater flexibility and guidance for the Center were adequately supported. However, recommendations for improved staff and more contracting work out were unsupported.