Data: The Good, The Bad, and The Ugly

You've heard the statisticians' joke? "There are outliers; and there are out-right liars." Okay, the humor is kind of lame. And it is probably as likely that you will hear Jay Z sing Brahms' Lullaby as wring humor from a number cruncher. No offense intended. It's only that I know who you are. The point is that in any given data set, such as scores from standardized student assessments, there are extremes.

Most scores that lie outside the norm are due to random variation. However and as we know, some scores may be attributed to manipulation, a kind of insider trading--usually perpetrated by adults--that inflates the average. That the outliers systematically inflate, as opposed to deflate, the average is the key detecting the crime. Just ask the good folk in Atlanta Public Schools.

I do not mean to pick on any one particular district, as there are many professionals who have fallen prey to the punitive pressures wrought by No Child Left Behind. APS one is but the most recent example. I will leave it to Diane Ravitch and her expose, The Death and Life of the Great American School System: How Testing and Choice Are Undermining Education, about which I have written before, to argue the case for reexamining how we gather, use, and too often abuse data in the name of helping children.

For the readers of this blog, I wish to make two points tonight. Point One, we need to remember that data by themselves not only say nothing (as in the oft-repeated phrase, "The data speak for themselves") but are, in fact, a kind of idol, standing for, but not actually being, the children in our care. Point Two, we need to make sure we educators are gathering the right data about our instructional choices. Collectively, the points constitute the good, the bad, and the ugly of data.

Let me begin with a confession relating to Point One. I hate "data-driven decision making." There, I said it. Yes, I know it's sexy and infectious (not in an STD way, mind you) and all the rage in capitals across the continent. And who doesn't want to be fashionable? I will also grant that those well-meaning people who use the term intend only that we best decide issues using data. But they don't say it that way.

In my mind, there is something insidious to the point of Orwellian to being driven by as opposed to driving. If data do not speak for themselves, then what is being said by whom on behalf of collected data that would make practitioners harness themselves to an unknown man's plow?

In the interest of being constructive, may I offer this simple verbal substitution: let us henceforth be "data-informed." If driving is to be involved, I'd rather be the driver than the driven. Who knows whose hands were last on that plow?

Seriously, I have seen more times than I can count programmatic over-correction and even abandonment because of the misinterpretation of student test scores. The best way to know if you are driving or being driven is to think to yourself: What else could these data mean? 

About Point Two, what are the right data to inform instructional choices? Victoria Bernhardt argues in Data Analysis for Continuous School Improvement that there are four areas in which we may collect data: demographics, perceptions, student learning, and school processes. We are familiar with and adroit at collecting the first three. The fourth, however, school processes, not only constitutes an area that we typically ignore but is, she argues, the one over which we have the greatest control.

Questions about and opportunities to collect data include: How do we group students for instruction and what impact does it have on achievement? What is the impact of the school bell schedule on learning? To what extent and with what impact are professional learning teams making a difference? School processes are the practices that make a school a school. Does it not make sense that the way the school organizes and operates daily be subject to a little investigation?

Bottom line: Leaders get to start conversations that make people think. I suggest that leaders use words carefully and remember why and who they serve. Data are no replacement for thinking.