When standardized tests measure the test, not the student

New York state's standardized education exams, like tests in Wisconsin and many other states, have been used to gauge the effectiveness of school reform efforts. The reforms that have been credited with improving test scores in New York City include mayoral control, charter schools, and teacher bonuses. But critics have long taken issue with these findings by raising questions about the tests themselves.

A recent in-depth story in the New York Times reveals that problems with the New York state tests have been a well-known secret for several years among city and state leaders. The tests themselves were recalibrated this year in order to address these problems and, as a result, achievement rates dropped dramatically. The tests will soon be redesigned altogether.

Says the Times:

New York has been a national model for how to carry out education reform, so its sudden decline in passing rates may be seen as a cautionary tale. The turnaround has also been a blow to Mayor Michael R. Bloomberg and his chancellor, Joel I. Klein, who despite warnings that a laserlike focus on raising scores could make them less and less reliable, lashed almost every aspect of its school system to them. Schools were graded on how much their scores rose and threatened with being closed if they did not. The scores dictated which students were promoted or left back, and which teachers and principals would receive bonuses. The test scores were even used for a new purpose this year: to help determine which teachers should receive tenure.

The specific problems with the New York tests are not present in the Wisconsin state tests. Wisconsin's exams are neither as short nor as narrowly focused as the New York exams, and the questions are not released publicly after the annual exam period. It is these facets that have made the New York exams easy to predict, and therefore easy to master. The result has been tests that do not accurately measure student learning, but in fact measure the tests' passability--an incredible 81% of all students in the state were deemed proficient in math and 69% were proficient in reading according to the 2008 test results. (The 2009 tests resulted in 84% of all New York City public schools receiving an A in the mayor’s grading system.)

While many educators, policymakers, and researchers were aware that the scores were too good to be true, the stakes seemed too high to reverse course. However, John King, New York’s deputy education commissioner, told the Times, “If people had known what an effective lever the tests would be of driving behavior, I think they would have designed the tests differently.”

This is a good lesson for Wisconsin as our state embarks on a process to design and adopt a new standardized testing scheme--tests must be designed with all their purposes in mind. To date there has not been a vigorous public debate in Wisconsin about the potential uses of the new tests. To what extent will they be used to measure teacher performance and determine teacher pay? Will they measure students' skills as well as content knowledge? Will they measure student performance in comparison to national norms or as a reflection of state standards? Will individual student growth be measured? Will the tests be used to measure the effectiveness of governance reforms, school choice, virtual schools, and/or charter schools?


Standardized tests do not paint a complete picture of student learning, but they are the easiest method of measuring and comparing student achievement and thus play an out-sized role in education policy. Awareness of standardized tests' limitations should underlie any policy or educational decisions that will be determined by them.