Program Evaluation

In Process

Although no agency GAO reviewed had a formal policy describing evaluation planning, all followed a generally similar model for developing and selecting evaluation proposals. Agencies usually planned an evaluation agenda over several months in the context of preparing spending plans for the coming fiscal year. Evaluation staff typically began by consulting with a variety of stakeholders to identify policy priorities and program concerns. Then with program office staff they identified the key questions and concerns and developed initial proposals. Generally, the agencies reviewed and selected proposals in two steps: develop ideas to obtain initial feedback from senior officials and develop full-scale evaluation proposals for review and approval. The four general criteria these mature agencies use to plan evaluations were remarkably similar: (1) strategic priorities representing major program or policy area concerns or new initiatives, (2) program-level problems or opportunities, (3) critical unanswered questions or evidence gaps, and (4) the feasibility of conducting a valid study. The agencies' procedures differed on some points. External parties' participation in evaluation planning may reflect these agencies' common reliance on nonfederal program partners. Only the offices GAO reviewed in HHS' Centers for Disease Control and Prevention held formal competitions to rank-order proposals before submitting them for approval; in other agencies, senior officials assessed proposals in a series of discussions. When evaluation authority and funds are tied to the program, evaluators generally choose not which programs to evaluate but which research questions to answer. Sometimes this resulted in a program's never being evaluated. Evaluation units at higher organizational levels conducted a wider range of analytic activities, consulted more formally with program offices, and had less control over approvals. The Congress influences an agency's program evaluation choices through legislating evaluation authority, mandating studies, making appropriations, and conducting oversight. GAO concludes that (1) all four criteria appear key to setting an effective evaluation agenda that provides credible, timely answers to important questions; (2) most agencies could probably apply the general model in which professional evaluators iteratively identify key questions in consultation with stakeholders and then scrutinize and vet research proposals; (3) agencies could adapt the model and decide where to locate evaluation units to meet their own organizational and financial circumstances and authorities; and (4) agencies' reaching out to key program and congressional stakeholders in advance of developing proposals could help ensure that their evaluations will be used effectively in management and legislative oversight. GAO makes no recommendations.