You say "accountability," I say "longitudinal study"

In a report released last week that did not get press other than a post on the education blog of the Journal Sentinel, the Legislative Audit Bureau rehashed the first-year findings of the School Choice Demonstration Project's study of the Milwaukee Parental Choice Program (MPCP). Interestingly, the Bureau found the study's data cannot provide information about performance in individual schools. In response, a prominent school choice lobbying group has called for test scores to be reported annually on a school-by-school basis.

The overall findings, released last February, were not as positive as education reform supporters had anticipated. The Audit Bureau re-analyzed the data and confirmed these findings. For example, the sample of choice students in the private schools had lower reading scores on state standardized tests than a matched sample of MPS students at three of six grade levels. At all six grade levels tested, the private school students scored lower than a random sample of MPS students. In nearly all cases, however, the differences were not statistically significant.

As the Demonstration Project's researchers emphasized, these results are to serve as a baseline. The real test will be whether scores improve in coming years and how those improvements compare across school types. The Audit Bureau also agrees that tracking score improvements over the years will be most important. But, the Bureau is worried that due to attrition within the samples, there may not be enough students in future years of the study for long-term comparisons to be reliable.

That is just one of many caveats in the Audit Bureau's report, most of which were methodological. The main concern of the auditors is the lack of usefulness of the study overall. Says the report:

While the project is designed to answer several academic research questions, there are limitations to its usefulness for policymakers... [W]e had initially believed that the project would provide us with data that identified the school attended by each Choice pupil who took the tests... [H]owever, citing confidentiality concerns, the project chose not to provide information on these pupils' scores... Because the project's data do not identify the Choice pupils and schools, we are limited in what we can report and confirm.

The auditors go on to say that this limitation means they cannot report information about academic performance specific to each choice school, despite there being such information available about every public school in Milwaukee and the rest of the state. The study, therefore, provides no accountability for individual private schools accepting taxpayer-funded vouchers in lieu of tuition.

While perhaps the Audit Bureau shouldn't be surprised that a longitudinal study is not a great vehicle for providing accountability, their words and tone indicate that they were expecting to be able to report out performance data for individual schools. Indeed, in early 2006 when the longitudinal study was passed as part of a compromise bill to lift the enrollment cap on the voucher program, it was cast as an accountability measure by many, including the editors of the Journal Sentinel (also here) and proponents of lifting the cap. In fact, it is still discussed under the heading of accountability on the School Choice Wisconsin website.

Yet, School Choice Wisconsin's official response to the audit report included the following language:

Raw test data released on a school-by-school basis are not meaningful and even can be misleading. Such data will not provide legislators or parents with useful information about the academic quality of individual private schools in the Milwaukee Parental Choice Program (MPCP).

The statement goes on to say:

School Choice Wisconsin favors an independent annual report available to Milwaukee parents that includes school-by-school testing data based on individual student progress. This report would use a common basis for reporting test scores at schools in the MPCP, the Milwaukee Public Schools, and at independent charter schools. We also favor an aggressive information campaign to inform parents about their options.

This second quote is the first call from School Choice Wisconsin for any kind of comparable reporting of test scores that could be used for accountability purposes. (Why they feel the Demonstration Project is not sufficiently independent to report school-by-school data is not clear from the press release; but that concern could be discussed and worked out.)

The important thing is in response to the Audit Bureau's concern that the longitudinal study has not released its findings on a school level, the most prominent school choice advocacy group is calling for comparable test score results to be reported. If legislators, educators, and advocates can come together around real accountability, then perhaps soon parents and taxpayers will be able to make informed decisions about school quality.