Noting that the disappointing findings of the first-year evaluation of the 21st-Century Community Learning Centers (21st CCLC) after-school program were offered as a rationale for a requested decrease in funding in President Bush’s fiscal year 2004 education budget, this report compiles expert commentary on methodological issues in that evaluation and discusses the implications of the findings for current policy as well as for future evaluation design, implementation, and use. Researchers, evaluators, and practitioners assert that evaluation needs to shift from a system of accountability only to one of learning for continuous improvement and accountability. Their comments are meant to stimulate a larger conversation about a productive research and evaluation agenda, about solutions to methodological problems, and about how to maximize the use of research and evaluation to support policy formation and service improvement. Serious methodological concerns are raised about the 21st CCLC program evaluation that call into question the findings of the report, including problems with ensuring that the middle school treatment and comparison samples were comparable prior to treatment, possible contamination of the middle school treatment and comparison groups, data collection on only half of the intended elementary district sites, and inclusion of programs in the elementary sample that had only an incidental focus on academic and developmental experiences for children. The report concludes with a discussion of approaches that program evaluators can take in the “new evaluation game” amid the five principles for scientifically based research in education put forth by the No Child Left Behind Act. (Contains 13 endnotes and a bibliography of additional commentary and other resources.) (KB)
