Study: Preschool rating systems disconnected from child outcomes

EXCERPTS:

Preschools that are highly ranked by state evaluation systems produce outcomes for children that are not significantly better than lower-ranked programs because those systems may be including too many indicators, according to a study released this month in the journal Science.

Researchers wanted to study the connection of student learning to Quality Rating and Improvement Systems, which have been created as a way to evaluate preschools and share those rankings with the public. The federal investment in the Race to the Top Early Learning Challenge grants, as well as funding from states and foundations, have prompted widespread adoption of these systems, which Education Week explored in an June article. Nearly every state has, or is developing, a QRIS.

But those systems may draw in so many elements that the resulting ranking may end up with a distant connection to teacher-child interactions, which are known to be a strong predictor of how well children do in preschool and afterwards, said Terri J. Sabol, a postdoctoral fellow at the Institute for Policy Research at Northwestern University in Evanston, Ill., and the study's lead author.

"My biggest takeaway is that states need to simplify their rating systems," Sabol said in an interview. "There's something really appealing about having these five-star systems, but that comes at a cost because those stars don't mean a lot for child outcomes."

The study used data collected in two studies conducted by the National Center for Early Development and Learning: the
Multi-State Study of Pre-Kindergarten that included six states, and the Statewide Early Education Programs Study that included five states. Together, the two studies provide detailed information on prekindergarten teachers, children, and classrooms in 11 states, collected between 2001 and 2004. (A snapshot of those studies' findings contain a wealth of information about pre-K programs.)

Researchers tried to replicate, as closely as possible, the scoring algorithms that states use in their QRIS rankings. All of the states measured preschools on staff qualifications, staff-child ratio and group size, family partnerships, and learning environments. The researchers also created an additional measure, teacher-child interactions, which was evaluated through the Classroom Assessment Scoring System, or CLASS. (CLASS has been adopted in the past few years by Head Start as a method of evaluating preschool quality.)

After linking outcomes to the evaluation measures, researchers found that teacher interactions had the highest connection to student learning, followed by learning environment. Teacher qualifications, class size, and family partnerships had a weaker and sometimes inconsistent connection. Thus, rating systems that combined all those measures also had a weaker and less consistent connection to child outcomes, the study showed.

Study co-author Robert C. Pianta, the dean of the education school at the University of Virginia and the creator of the CLASS evaluation instrument, said that one way to simplify rating systems could be to make some elements non-negotiable. "There shouldn't be any variation in [teacher-child] ratio, or health or safety provisions, or whether the teacher has certain level of training," he said. Once those elements are removed from the rating systems, the systems can focus more closely on the most powerful measures, he said.

"The people doing this work are terrific, they're very knowledgeable about the field," Pianta said, but he added that the desire to include many different measurements is a challenge. "We're really rolling out a big policy without knowing the consequences of that policy might or might not be."

-reprinted from Education Week