I love what I do.

I love technology and education. Maybe you like knitting. Cool. My thing is educational technology.

Please feel free to leave a comment...

Major new edtech report. Some technology doesn’t help math and reading

Friday, April 06, 2007

The United States Department of Education has released a major new study - click here for study (Warning: 2.4 MB PDF!).

The essential message of the study:

Test scores were not significantly higher in classrooms using selected reading and mathematics software products. Test scores in treatment classrooms that were randomly assigned to use products did not differ from test scores in control classroom by statistically significant margins.

Effects were correlated with some classroom and school characteristics. For reading products, effects on overall test scores were correlated with the student-teacher ratio in first grade classrooms and with the amount of time that products were used in fourth grade classrooms.  For math products, effects were uncorrelated with classroom and school characteristics.

Here is my initial reactions (I’ve spent about 30 minutes skimming):

1) Don’t attack the study design - it’s actually a very well-designed study, and these results are probably accurate. I’m always suspicious when people attack the experimental design - I like to examine the findings, which are often askance from the data.

These are first year results - this is ongoing, so we’ll see next year what the data reveals.

2) In my opinion, the correct question the ed-tech community should be asking now is: “How can we make reading and math educational technology better?” So far, I haven’t seen what made the software ineffective - that would be a good starting place.

3) This study (understandably) addresses a relatively narrow field of software practices. I think we should be sensitive to the “don’t throw the baby out with the bath water” phenomena”.

This is the software used. This is, by the way, a veritable who’s-who of educational software.

From the first grade reading camp:

Riverdeep (product)
Pearson Digital Learning (product)
Headsprout (product)
Plato Learning (product)
Autoskill (product)

From the fourth grade reading group:

Leaptrack (product)
Scholastic (product)
Pearson (product)

From the math group:

Houghton Mifflin (product)
Plato Learning (product)
iLearn (product)

From the algebra group:

Carnegie Learning (product)
Plato Learning (product)
Larson Learning (product)


On 06 April 2007, Doug Holton inscribed the following thoughts about this post:

That’s a good summary.  You’re right, it’s best not to simply dismiss the study because of its design.  Conversely, we shouldn’t take it at face value and simply claim educational software is a failure.

It showed these software products on average aren’t leading to any significant difference in student’s learning.  But that is only in the first year of using them.  I’m guessing the teachers didn’t alter their normal teaching practices and curriculum very much, so it would make sense that student learning wasn’t impacted much (or hurt in some cases I saw in their graphs - like carnegie’s cognitive tutoring software).

Also, I haven’t read the details of the types of schools and students, but I think some software products like these work better with certain situations.  For example, helping reinforce basic reading skills vs. tutoring higher level reasoning abilities.  Random assignment of the software to different classes may lead to some mismatches between what the software is ideally to do and who receives it.

Perhaps we need some kind of tool that guides teachers in selecting the best software to address the specific needs of their students, rather than a site that merely rates the best software overall.

Commenting is not available in this channel entry.


Bill MacKenty, Chief Zuccini

I make a difference in the life of kids. You want to tell me what's more rewarding?



This is my full resume. It has all my work experience since I graduated from college in 1992, including certifications, professional memberships, and descriptions of my work.


Polish Resume

This is my full resume translated into Polish. My wife tells me it is a literal translation, and as such might convey a slightly different meaning to Polish speakers.