You say "accountability," I say "longitudinal study"

In a report released last week that did not get press other than a post on the education blog of the Journal Sentinel, the Legislative Audit Bureau rehashed the first-year findings of the School Choice Demonstration Project's study of the Milwaukee Parental Choice Program (MPCP). Interestingly, the Bureau found the study's data cannot provide information about performance in individual schools. In response, a prominent school choice lobbying group has called for test scores to be reported annually on a school-by-school basis.

The overall findings, released last February, were not as positive as education reform supporters had anticipated. The Audit Bureau re-analyzed the data and confirmed these findings. For example, the sample of choice students in the private schools had lower reading scores on state standardized tests than a matched sample of MPS students at three of six grade levels. At all six grade levels tested, the private school students scored lower than a random sample of MPS students. In nearly all cases, however, the differences were not statistically significant.

As the Demonstration Project's researchers emphasized, these results are to serve as a baseline. The real test will be whether scores improve in coming years and how those improvements compare across school types. The Audit Bureau also agrees that tracking score improvements over the years will be most important. But, the Bureau is worried that due to attrition within the samples, there may not be enough students in future years of the study for long-term comparisons to be reliable.

That is just one of many caveats in the Audit Bureau's report, most of which were methodological. The main concern of the auditors is the lack of usefulness of the study overall. Says the report:

While the project is designed to answer several academic research questions, there are limitations to its usefulness for policymakers... [W]e had initially believed that the project would provide us with data that identified the school attended by each Choice pupil who took the tests... [H]owever, citing confidentiality concerns, the project chose not to provide information on these pupils' scores... Because the project's data do not identify the Choice pupils and schools, we are limited in what we can report and confirm.

The auditors go on to say that this limitation means they cannot report information about academic performance specific to each choice school, despite there being such information available about every public school in Milwaukee and the rest of the state. The study, therefore, provides no accountability for individual private schools accepting taxpayer-funded vouchers in lieu of tuition.

While perhaps the Audit Bureau shouldn't be surprised that a longitudinal study is not a great vehicle for providing accountability, their words and tone indicate that they were expecting to be able to report out performance data for individual schools. Indeed, in early 2006 when the longitudinal study was passed as part of a compromise bill to lift the enrollment cap on the voucher program, it was cast as an accountability measure by many, including the editors of the Journal Sentinel (also here) and proponents of lifting the cap. In fact, it is still discussed under the heading of accountability on the School Choice Wisconsin website.

Yet, School Choice Wisconsin's official response to the audit report included the following language:

Raw test data released on a school-by-school basis are not meaningful and even can be misleading. Such data will not provide legislators or parents with useful information about the academic quality of individual private schools in the Milwaukee Parental Choice Program (MPCP).

The statement goes on to say:

School Choice Wisconsin favors an independent annual report available to Milwaukee parents that includes school-by-school testing data based on individual student progress. This report would use a common basis for reporting test scores at schools in the MPCP, the Milwaukee Public Schools, and at independent charter schools. We also favor an aggressive information campaign to inform parents about their options.

This second quote is the first call from School Choice Wisconsin for any kind of comparable reporting of test scores that could be used for accountability purposes. (Why they feel the Demonstration Project is not sufficiently independent to report school-by-school data is not clear from the press release; but that concern could be discussed and worked out.)

The important thing is in response to the Audit Bureau's concern that the longitudinal study has not released its findings on a school level, the most prominent school choice advocacy group is calling for comparable test score results to be reported. If legislators, educators, and advocates can come together around real accountability, then perhaps soon parents and taxpayers will be able to make informed decisions about school quality.

No Worker Left Behind: Testing job applicants not shown to harm equity

UWM economist Marc Levine’s most recent figures (2006 data) on what he has termed “the crisis of black male joblessness in Milwaukee,” identifying that 46.8 percent of working-age black males in the city are out of work, suggest that Milwaukeeans should take notice of studies about minorities and employment. A new study in the Quarterly Journal of Economics by David Autor and David Scarborough makes a connection between equitable hiring and worker productivity.

With employers increasingly using company-designed standardized tests to measure applicants’ potential job skills, Autor and Scarborough wanted to find out what effect this testing had on rates of minority hires. Perhaps testing would reduce racial discrimination (and increase minority employment) by introducing objective data to rely on in hiring. Due to multiple societal and demographic factors, overall, minorities as a group tend to score lower on standardized tests than non-minorities. So, would the testing lead employers to hire fewer minorities based on differential scores? Additionally, would the employees hired based on test scores perform better than those hired before the firms introduced testing?

The researchers used data from over 1,300 retail stores of a national chain, and determined the test in use to be non-racially-biased. After examining hiring and job tenure both before and after testing was instituted, the study found that employees hired using testing had higher job tenures by 10%. Though minorities did score lower than others on the standardized test, minority hiring was unaffected by the introduction of testing to the application process. Furthermore, the aspects of testing that enhanced employee productivity accrued to both minority and non-minority job applicants.

The testing not affecting minority employment is good news, but the increase in job tenure accruing to both minorities and non-minorities is even more encouraging. It implies that the job testing allowed employers to make more informed decisions of whom to hire in both the minority and non-minority pools of applicants.

The study’s findings, as some have noted -- especially the fact that minority hiring was stable despite lower test scores – suggest that testing applicants is not incompatible with affirmative action goals.

The Wire's lost sixth season: Pothole crews

The realism of HBO's gritty series set in various Baltimore institutions is often debated. One nuance that didn't miss the producers of The Wire: the intense biweekly CitiStat meetings in which a city department head is in the hotseat as the mayor and his staff comb through reams of departmental performance data. (See episodes three and ten of season three for scenes set in internal police department stats meetings.)

CitiStat is a city management system consisting of biweekly meetings of the mayor and his stats team with agency and department heads. These meetings focus on a department's weekly performance on a set of metrics aimed at measuring timeliness and quality of service to citizens. Performance is tracked over time and drives policy and budgetary decisions.

To illustrate: Suppose Baltimore had a horrible winter that wreaked havoc on local roads and resulted in numerous potholes. (Sound familiar?) The CitiStat team would be on top of it. Each week the department head reports to the team how many complaints about potholes were received (the city has a non-emergency service request 3-1-1 phone number), how many of those particular holes were fixed within 48 hours, how many total potholes were filled, how many crew members worked, and how many crew-days were worked. The information is reported by sector of the city, so that if particular neighborhoods get better service than others, it can be fixed before becoming a scandal.

Since the city has been tracking this data since 2002, they also know whether these figures are above or below average for that week of the year, allowing them to adjust their budgeted figures for pothole filling as the year goes on. (Personnel data is also tracked...how many employees were absent, how many worked overtime, etc.)

Now imagine this type of data being available and used to drive decisionmaking for every city service, including public safety. In Baltimore it has resulted in cost savings varying from energy savings due to more efficient lighting patterns to more stray animal adoptions to more miles of street sweeping and graffiti removal per year.

The tenets of the CitiStat process are four-fold:

-Accurate and timely intelligence shared by all,
-Rapid deployment of resources,
-Effective tactics and strategies, and
-Relentless follow-up and assessment.

Other cities that have adopted the data-based management system include Providence, RI; Buffalo, NY; Atlanta, GA; and Somerville, MA. Not all have found as much success as Baltimore. The system has been around long enough now that there is research to tell us how and why some cities have succeeded and others have not. The seven most important factors include:

1. Having a clear purpose--It's not enough just to collect data; there must be a specific result in mind. Eliminating potholes, for example. A general "safer streets" goal might not be enough. In addition, it's tempting to measure things just because you can and its fun to go data mining in pursuit of interesting nuggets. But the goals should drive the data collection, not vice versa, and all measurement should be done with the purpose in mind.

2. Ensuring someone takes responsibility for achieving the purpose--Is there just one person in charge of safer streets? Probably not. It's much more likely that someone's neck is on the line when it comes to potholes.

3. Meeting regularly and frequently with the stat team--If data are collected on a daily basis, then progress should be reviewed frequently enough (biweekly, for example) that problems can be headed-off or progress can be praised.

4. Running meetings with authority--Since it's unlikely a mayor can devote the time to running every meeting, authority must rest with whomever does run the meetings, so that deficiencies, if found, can be followed up on. Consistency is also important so that over time, the leader of the meeting has a feel for the depth of the problems, the growth of the progress, or the appropriateness of the goals.

5. Having a dedicated analytical staff--Every local government is capable of producing data in spades...making meaning of that data is what's most time- and resource- consuming. For a CitiStat program to work, there must be a stat team to analyze data full-time.

6. Relentlessly following-up--Department heads must ensure that their staff is prepared for each meeting, understands the purpose, has a game plan for getting results, and has made improvements since the last meeting.

7. Understanding constant conflict is not productive, nor is coddling--Baltimore's CitiStat meetings have a reputation of being brutal and combative games of "gotcha." Somerville takes a different tactic by providing department heads with an agenda the day before, so they can come prepared with explanations or solutions. Meetings should not be punitive, but should not be mere show-and-tell exercises, either.

It must be noted, however, that there is a real risk when using a data-driven management system such as CitiStat to make policy decisions: forgetting that numbers don't always tell the real story. In fact, one episode of The Wire illustrates this risk beautifully: When one police commander allows unhampered drug trading on one street, the number of felony incidents in the neighborhood drops dramatically. The resulting "mini Amsterdam" is seen as illegal, insane, and yet brilliant--a 14% reduction in major crime almost overnight. To a commander under intense pressure to improve the stats, the ends justify the means.

For policy wonks or civil servants, a CitiStat season of The Wire, showing heads rolling while hot asphalt is steaming, would have been must-see TV.

Transportation on my mind...

Before heading over to Marquette University Law School yesterday to watch Mayor Barrett and County Executive Walker debate Milwaukee's transit future, I came across three other transportation stories making news that day.

The first, in the Philadelphia Inquirer, headlined a call from the I-95 Corridor Coalition , made up of east coast transportation and police officials, for a doubling of highway spending and "drastically" increased use of transit and rail in the corridor extending from Maine to Florida. Interestingly, a coalition spokesman acknowledged that neither the states nor feds have the capacity to fund the estimated $71 billion annual cost and argued that public-private partnerships be utilized to help fund improvements. That is the direction in which the Pennsylvania Legislature may be headed in light of proposals to lease the Pennsylvania Turnpike and authorize privately built toll lanes on existing highways.

The second and third articles, from the Associated Press and Wall Street Journal, detailed the "surge" in transit ridership nationwide during the second quarter and the significant challenges facing many transit agencies as they try to accommodate it. The AP story laid out the arguments for and against increased federal funding, while the Journal story reported on specific proposals in Congress to boost mass transit funding.

Armed with this context, I listened to the mayor and county executive lay out their respective transportation visions and argue again about use of the $91.5 million in federal funds authorized 17 years ago for a Milwaukee transit project. As I did so, I couldn't help but reflect on the following:

• While we continue to argue with each other on the local level, other mega-regions have formed powerful coalitions to advocate in Washington for their collective transportation needs. Who is more likely to get their fair share of an insufficient federal transportation funding pie, huge regions of the country who band together to fight for their parochial interests, or mid-sized metropolitan areas whose elected officials can't even agree on priorities among themselves?


• Not only Pennsylvania, but countless other states are acknowledging that their transportation infrastructure needs and those of the nation as a whole are so staggering that non-public funding and/or operation of parts of that infrastructure, as well as congestion pricing or other tolling mechanisms, must be contemplated as at least part of the solution. Ironically, the decrease in driving and popularity of smaller vehicles is making the problem even more acute, as gasoline tax revenue is no longer an elastic source of revenue. Is Wisconsin behind the eight ball in awakening to these realities?


• The Journal article notes that "momentum is building in Congress" to increase funding for public transportation, signaling good news for those counting on greater federal support to build and operate light rail, bus rapid transit and/or commuter rail in southeast Wisconsin. At the same time, however, both that article and the AP story describe the monumental challenges facing transit systems in paying for existing bus and rail service. That reality - combined with a depleted Federal Highway Fund that has some in Washington talking about diverting transit dollars for highway needs - reflects the challenges Milwaukee will face in attempting to obtain federal money for new transit services.

Of course, the fundamental lesson here is that transportation needs not only here in southeast Wisconsin, but across the country, are immense, and that other states and metro areas are objectively assessing those needs and developing strategic, diversified and cohesive approaches to meeting them. If indeed Washington is poised to provide more money, then it's a pretty safe bet that those with the best plans and the most unity will be first in line to get it.