Education Research

"Some people expect educational research to be like a group of engineers working on the fastest, cheapest, and safest way of traveling to Chicago, when in fact it is a bunch of people arguing about whether to go to Chicago or St. Louis." -- Gene Glass, former president of the American Educational Research Association

"Unfortunately, too much of what we recognize as education research is simply opinion buttressed by anecdotes." -- U.S. Representative Michael Castle

"Research has shown..."

Hoo-boy -- When they start hauling out that phrase "research has shown" it's time to pull on the hip boots. Perhaps the single greatest accomplishment of progressive education is development of that phrase.

(It might be a good research project to determine if educrats say "Research has shown" even more often than "You are the only one who has ever complained.")

The wackiest fads are justified simply by invoking the magic words, "research has shown." It completely stops discussion in its tracks by eliminating any argument.

When you hear "research has shown..." how can you know if the claim being made true? Is there any real research supporting the changes being imposed on your school?

Here are links to articles about the state of EDUCATION RESEARCH:


• Scientifically Based Research -- U.S. Department of Education: The new federal education law, the "No Child Left Behind Act," demands in many places that programs funded by federal dollars be supported by "scientifically based research." This is a big step for many education leaders who are instead very accustomed to practices based on pure theory, and are thus unclear on this new and more scientifically supportable approach. To clarify what "scientifically based research" is and to explain why it is so crucial, the U.S. Department of Education has created a website collecting reports and conference transcripts, relating both to education in general and to specific content areas.

• Education Policy and Information: We Need Good Science To Analyze and Interpret Educational Data by E. D. Hirsch, Jr. Excerpts: "To those who must make decisions, it is disconcerting that so much educational data have produced so little usable information. ... In a field beset with ideology and politics, it is not surprising that good science is in short supply. But regardless of one's faith in markets or, alternatively, in state regulation, we still need reliable information, which means that we need really good science, not the impostor that now calls itself 'research' in the field of education."

• Classroom Research and Cargo Cults by E.D. Hirsch Jr., Policy Review, October 2002. Prof. Hirsch continues asking why decades of supposed "research" provides so little useful guidance for setting school policy. The title refers to an observation by Nobel-winning physicist Richard Feynman, and Hirsch's uses that to explain that "while educational research sometimes adopts the outward form of science, it does not burrow to its essence." This article provides rich detail on the problems in what is currently published as "education research."

• Nice Theory ... Any Proof? This article discusses a recent article about movements towards "evidence-based" social policy, "by promoting randomized control group experiments to inform policy-making. Such experiments are required in medicine, but amazingly rare in social policy. ... A case in point is the 'whole-language' approach to reading, which swept much of the English-speaking world in the 1970s and 1980s. ... Unfortunately, the educational theorists who pushed the whole-language notion so successfully did not wait for evidence from controlled randomized trials before advancing their claims. Had they done so, they might have concluded ... that effective reading instruction requires phonics."

• Education Research Is Under The Microscope by Joshua Benton, Dallas Morning News, Monday, July 29, 2002. Excerpts: "Education research has been a punching bag for decades -- largely for good reason. ... Anecdotes and hearsay are often as prominent as hard data. And some scholars, motivated by ideology, seem to reach their conclusions before the research even starts."

• "Why Education Experts Resist Effective Practices (And What It Would Take to Make Education More Like Medicine)" by Douglas Carnine. You can view the html version, or download a PDF version.
  
  
• Research and innovation: Let the buyer beware by J. E. Stone and A. Clements, in "The Superintendent of the Future," edited by Robert R. Spillane & Paul Regnier. "Schools are inundated with research that promises to improve achievement. Yet when programs are implemented results always seem to fall short. How can it be in school after school, year after year? The answer depends on whom you ask."
  
  
• Sciencephobia: Why education rejects randomized experiments by Thomas D. Cook. "The American education system ... has been continually roiled by tides of ... experimentation ... Experimentation, however, means more than simply changing the way we do things. It also means systematically evaluating these alternatives."
  
  
• Detriment to Students: Education reformers take aim at flawed, incomplete research by Susan Laccetti Meyers, Atlanta Journal-Constitution, October 3, 2001. "...One of the real impediments to true education reform is that many of the scholars either have preconceived ideas that they attempt to validate, or they have a predetermined agenda. Many researchers are funded by groups or liberal-leaning foundations that have not warmed to reform. Many, too, are based at colleges of education and are loath to venture outside their personal experiences and training."
  
  
• The Difference Between Qualitative and Quantitative Research, by George K. Cunningham, professor, Department of Educational and Counseling Psychology University of Louisville, August 29, 1999. "I have to respond to this often-repeated assertion that there is extensive empirical research in support of constructivist methods of instruction. When most people read this assertion, they assume that there are a series of studies in which groups of students are taught using different methodologies and that on the basis of some sort of objective measure, those taught using constructivist methods scored higher. Nothing could be further from the truth. Advocates of constructivism view such research methodology as a tool of a patriarchal system bent on subjugating the disadvantaged."
  
  
• Education Research and Evaluation and Student Achievement: Quality Counts by Dr. G. Reid Lyon, Chief, Child Development and Behavior Branch, National Institute of Child Health and Human Development, National Institutes of Health. This is his statement to the Committee on Education and the Workforce, U.S. House of Representatives, Washington, D.C., May 4, 2000
  
  
• The Virtues of Randomness by Robert Boruch, Education Next, Fall 2002. "Randomized field trials are a sturdy method of generating defensible evidence about the relative effectiveness of various interventions ... Yet this powerful technique of discovering what works has been slow to come to the field of education.
  
  
• Resisting the Assault on Science: The Case for Evidence-Based Reasoning in Educational Research (PDF file) by Richard E. Mayer
  
  
• Science, Art, and the Predispositions of Educational Researchers (PDF file) by Tom Barone
  
  
• "Experiments On Johnny's Brain: Experts Doubt Value Of 'Qualitative' Ed Research", Investors Business Daily, April 27, 1999. Excerpts: "'Among the community of people who have backing in scientific methods and use objective principles, [there is] a real difference in standards (between) education research that appears in education journals and what appears in psychology and science journals,' said Louisa Moats, project director for the Washington, D.C., site of the Early Interventions Project at the National Institute of Child Health and Human Development. ... 'University schools of education are held in low regard by others in the arts and sciences,' Moats said. ... 'There is no relation between what is practiced in a classroom and what is validated,' Moats said. 'It is unusual for consumers in education to decide what they are going to do and what materials they are going to buy based on independent, scientific validation.'"
  
  
• Research Bill Clears House, Education Week, May 8, 2002: "Unfortunately, too much of what we recognize as education research is simply opinion buttressed by anecdotes," says U.S. Representative Michael Castle of Delaware, sponsor of legislation creating a new "Academy of Education Sciences."
  
  
• Resisting Education's Fads by Gail Russell Chaddock, Christian Science Monitor, August 25, 1998. "Teachers call it the 'reform du jour,' and for many, it's the biggest challenge at the start of any school year. That's when the latest idea for how to improve student performance kicks in. In the 1970s, it was the open classroom, which knocked out the walls between classes to create flexible space and make learning more fun. Soon, carpenters were tapping away at new walls to get the noise level back down. In the 1980s, many districts tucked away the phonics books to make way for 'whole-language' instruction, which emphasized context and the personal value of reading. The new books were engaging, but many kids weren't learning to read. Teachers were ordered to dig out the flash cards. The 1990s brought down new mandates to teach to individual 'learning styles' -- despite a lack of consensus on how to measure learning styles, or whether it is better to teach to a learning style or to help students overcome it. Even critics note these ideas have valid points. But they were often adopted without data -- without balancing the claims of competing teaching techniques -- and then taken to extremes."

• Sidebar: How To Fad-Proof Your School by Gail Russell Chaddock, Christian Science Monitor, August 25, 1998. "The key to fad-proofing your school is to look for things that work and avoid those that don't. Here are suggestions from some top superintendents and teachers."

  The Dalmatian and Its Spots by Siegfried Engelmann, Education Week, January 28, 2004. "At least part of the problem educators have in establishing effective instruction has to do with the illogical recommendations that researchers make. ... This illogical practice is the confusion about what follows from a true statement. Here's a noneducational example: If a dog is a Dalmatian, it has spots. Therefore, if a dog has spots, it is a Dalmatian. The first statement is true. The second statement doesn't follow from the first. The probable response from most readers is that nobody could be naive enough not to recognize this flaw. ... Unfortunately, there are many educational parallels to the argument that all dogs with spots are Dalmatians."

• The Education Research We Need? by Frederick M. Hess & Francesca Lowe, National Review Online, April 12, 2007. "Those eager to familiarize themselves more broadly with 'peer-reviewed' education scholarship had an exceptional opportunity earlier this month in Chicago when the American Educational Research Association (AERA) gathered for its annual conference. More than 12,000 scholars convened to consider a wealth of papers and analyses that have been vetted and approved by their peers. ... For instance, those policymakers interested in Creolist perspectives on social change, the hegemony of standard English, and the current status of the Whig party, doubtless flocked to the session featuring the research 'Beyond the Anglicist and the Creolist Debate and Toward Social Change,' 'The Ebonics Phenomenon, Language Planning, and the Hegemony of Standard English,' and 'The Whig Party Don't Exist in My Hood: Knowledge, Reality, and Education in the Hip-Hop Nation.' Now, while we may all be immigrants to the hip-hop nation, those who are interested in the import of immigration for instruction or assessment will surely want to catch the session 'Asserting Silenced Voices in Policy Debates Over (Im)migrants and (Im)migration,' featuring analyses like 'Exposing Contradictions: Students Talk Back to Discourses on (Im)migration' and 'Immigrant Identity and Racial Formations: A LatCrit Theoretical Analysis Case Study.' ... [Others] could attend 'Depending on How You Read It: The Trans-Contextualization of Pedagogies of Everyday Practice Across Urban Communities.' Offering much more than glitzy 'trans-contextualization of pedagogies,' the research included probing examinations of elderly African-American bridge players and everyday pedagogies in papers, like 'Identity, Positioning, Knowledge, and Rhetoric in the Pedagogical Practices of Elderly African-American Bridge Players' and 'Everyday Pedagogies in Basketball, Track, and Dominoes: Culture, Identity, and Opportunities for Competence.' ...
       "If their work is to command the respect and attention it deserves, serious education scholars need to invest more energy in policing the work that enjoys their imprimatur. So long as it is swamped by sophistic(ated) parentheses and 'Creolist perspectives,' it's going to be an uphill struggle for quality education research to have the impact it seeks or deserves."

  What Works Clearinghouse

• What Works Clearinghouse, U.S. Department of Education.
  

• Determining What Works in the Classroom by Paul Clopton and Bill Evers, San Diego Union Tribune September 8, 2004. Excerpts: "Teachers would like to know what teaching practices and instructional materials would work best for their students. Every teacher has heard the phrases 'studies show that ...' and 'research-based program,' but all too often teachers can't find usable research for their K-12 classrooms. ... The importance of high-quality research ... has prompted the U.S. Department of Education to provide a screening service in the form of the new What Works Clearinghouse web site. The Clearinghouse chose peer-assisted learning (also known as 'group learning') and middle-school mathematics as the first topics to investigate. So far, the results confirm the perception that we have a long way to go in the scientific study of education. For the peer-learning category, 15 of 191 studies met Clearinghouse criteria. (Some 109 studies have yet to be screened.) For the mathematics category, one met Clearinghouse criteria, one was close (met with reservations), and 48 did not meet the standards (20 are yet to be screened). Worse yet, neither of the two acceptable mathematics studies showed any significant difference between methods. Some argue that the standards used to evaluate studies are too stringent. ... If anything, the Clearinghouse may have engaged in a bit of grade inflation so that it didn't end up rejecting all the studies. ... Teachers need educational research that provides good advice on which materials and methods are best for classroom use for specified academic objectives with various groups of students. Clearly, the message from the What Works Clearinghouse is that educational research has a long way to go."
  
  

Project Follow-Through

• Our Failure To Follow Through by Billy Tashman, New York Newsday, November 15, 1994. "Project Follow Through, America's longest, costliest and perhaps, most significant study of public school teaching methods quietly concluded this year. The good news is that after 26 years, nearly a billion dollars, and mountains of data, we now know which are the most effective instructional tools. The bad news is that the education world couldn't care less. ... It may come as a shock to the layperson, but school policymakers haven't adopted Direct Instruction because they have an aversion to scientific research. Educators throw their weight behind the latest fad, then refuse to abandon it when it doesn't work. ... Follow Through demonstrated that scientific research and the classroom are still strangers to one another. Until they join forces, American schoolchildren will continue to receive a second-class education."

• Educators Ignore Proven Method of Improving Students' Learning by David Ziffer, School Reform News, October 1, 2006. David Ziffer is a school choice activist and one of the original co-founders of the Illinois Loop. He operated a Direct Instruction-based after-school reading program from 1997 through 2002. He has no current financial interest in Direct Instruction. Excerpts:
       "Imagine for a moment that we have a cheap cure for cancer. Now imagine the medical establishment is doing everything possible to discredit the cure and prevent its use, so doctors who treat (but don't cure) cancer can keep their jobs. Imagine millions of patients continuing to suffer and die because nobody--including most doctors--knows about the cure.
       "This is a picture of something that's really happening in education. But to understand our almost-unknown educational "cure," you have to know about Project Follow Through (PFT).
       "PFT is the world's largest-ever education research project, conducted between 1967 and 1977 by the U.S. Department of Education. Its results indicate there is a replicable, systematic curriculum that dramatically improves the quality of education in poor urban schools--one that can raise poor urban students' test scores to suburban levels. ...
       "PFT researchers selected 180 low-income urban and rural school districts nationwide in which elementary school performance was at approximately the 20th percentile. Nine different educational models were each allocated to some schools in approximately 20 districts, with the remaining schools functioning as controls.
       "Students were pre-tested to determine initial performance differences between PFT-model schools and control schools, so the final analysis could compensate for initial variances. At the conclusion of the project, two independent agencies were hired to collect and analyze the data.
  

  "DI even outperformed the constructivist models in areas in which they were supposed to excel."

       "The results were shocking ...
       "Of PFT's nine curriculum models, five were firmly constructivist and three were indeterminate. Only one model--Direct Instruction (DI)--firmly embraced the idea of teaching basic skills. Nobody was more surprised than the constructivist curriculum authors when PFT demonstrated two things.
       "First, the basic-skills-oriented DI far outperformed both the control groups and the other models.
       "Second, the five constructivist-style curricula actually reduced school performance in districts that were already among the lowest performers nationwide.
       "DI even outperformed the constructivist models in areas in which they were supposed to excel."
  Click here to read the full article's recap of Project Follow-Through.

• Comments on Curriculum and Project Follow Through, by Chris Jenner, member of District 26 school board, Cary, Illinois, December 19, 2005.

• "Follow through: Why didn't we? -- Are proven teaching methods already available?" by Cathy L. Watkins. "What do we do with a teaching technique that works? Surely, educators would welcome such a breakthrough with open arms. Incredibly, they haven't. Project Follow Through, the largest experiment ever undertaken to find effective methods for teaching disadvantaged children, discovered such a teaching method at a [research] cost of nearly a billion dollars." This article is available as a web page or as a PDF document.
  
  
• Project Follow Through: In-depth and Beyond by Gary Adams. Excerpt: "The Follow Through project was the largest, most expensive educational experiment ever conducted. This federal program was originally designed to be a service-oriented project similar to Head Start. However, because of funding cutbacks the emphasis was shifted from service to program evaluation. Over 75,000 low income children in 170 communities were involved in this massive project designed to evaluate different approaches to educating economically disadvantaged students from kindergarten through grade 3. State, school, and national officials nominated school districts that had high numbers of economically disadvantaged students. Parent representatives of these school districts chose to participate after hearing presentations from the 20 different program designers (sponsors). Each participating district implemented the selected sponsor's approach in one or more schools. For participating, each district received $750 per student beyond the normal level of funding.
  
  
• Research Regarding Direct Instruction by Dave Ziffer. Also includes information about Project Follow-Through.
  
  
• The Story Behind Project Follow Through by Bonnie Grossen, Effective School Practices, Winter 1995-6. Honest Follow-Through Needed On This Project by Marian Kester Coombs, Washington Times, Mar 24, 1998. "What if the federal government spent $1 billion over nearly three decades to study thoroughly the question of which teaching method best instills knowledge, cognitive skills and positive self-concept in students? What if that study were able to conclude exactly which method best does all three? Wouldn't the American people like to know about it? ... Direct instruction, one of the basic skills approaches, showed the greatest positive impact on all three types of development (see graphs).

• What the Data Really Show: Direct Instruction Really Works! by Jeff Lindsay

• The Ultimate Rejection of Follow Through Data by Siegfried Engelmann, 2006

When "Research Shows..."

"Studies Prove..." - Part 1 by Thomas Sowell, August 9, 2006. "Whenever I hear the phrase 'studies prove' this or that, it makes me think back to the beginning of my career as an economist at the Labor Department in Washington. ... It was a valuable experience so early in my career to learn that what 'studies prove' is often whatever those who did the studies wanted to prove. Labor Department studies 'prove' whatever serves the interest of the Labor Department, just as Agriculture Department studies 'prove' whatever serves the Department of Agriculture's interests."

"Studies Prove..." - Part 2 by Thomas Sowell, August 10, 2006. "My late mentor, Nobel Prize-winning economist George Stigler, used to say that it could be very instructive to spend a few hours in a library checking up on studies that had been cited. When I began doing that, I found it not only instructive but disillusioning. A footnote in a textbook on labor economics cited six studies to back up a conclusion it reached. But, after I went to the library and looked at those six studies, it turned out that they each cited some other study -- the same other study in all six cases. Now that the six studies had shrunk to one, I got that one study -- and found that it was a study of a very different situation from the one discussed in the labor economics textbook."

"Studies Prove..." - Part 3 by Thomas Sowell, August 11, 2006. "Often we hear that 'all the experts agree' that A is better than B or that 'studies prove' A to be better than B. But one of the reasons for this can be that only people who favor A over B are likely to get the money to conduct studies or be given access to the data needed for a study. When only people with one set of views are allowed to do certain studies, do not be surprised if 'studies prove' that set of views is right."

Quotes

From our page on education quotations, here are the entries on education research:

"It's ironic that we deny a brain-tumor patient with six months to live the choice of a promising new therapy because it's 'unproven,' but we'll let any zealous band of ninth-grade math teachers cripple 100,000 children for life by testing a pedagogical fad whose benefits are purely conjectural. No 'informed consent' forms are required for schools to experiment on kids."
-- Peter Pearson, Aptos, Calif., in the Chicago Tribune

"One accurate measurement is worth a thousand expert opinions."
-- Adm. Grace Hopper

"The fundamental principle of science, the definition almost, is this: the sole test of the validity of any idea is experiment."
-- Richard P. Feynman

"If it can't be expressed in figures, it is not science; it is opinion."
-- Robert A. Heinlein

"When you can measure what you are speaking about, and express it in numbers, you know something about it; but when you cannot measure it, when you cannot express it in numbers, your knowledge is of a meagre and unsatisfactory kind; it may be the beginning of knowledge, but you have scarcely in your thoughts advanced to the stage of science."
-- Lord Kelvin

"Unfortunately, too much of what we recognize as education research is simply opinion buttressed by anecdotes"
-- U.S. Representative Michael Castle of Delaware, sponsor of legislation creating a new "Academy of Education Sciences."

"Some people expect educational research to be like a group of engineers working on the fastest, cheapest, and safest way of traveling to Chicago, when in fact it is a bunch of people arguing about whether to go to Chicago or St. Louis."
-- Gene Glass, former president of the American Educational Research Association

"Educators have tended to treat fervently-held opinion with the same reverence as scientific fact for so long that they have lost the ability to discern the difference!"
-- Charles M. Richardson

"When the grand pooh-bah PhDs of education stand up and blow, they speak with great confidence about theories of teaching, and considering the test results, the bums ought to be thrown out."
-- Garrison Keillor "The trouble with educational experiments is that they all work."
-- unknown

"Research in education is not taken seriously by any other college or department on a university campus outside the colleges of education. It's very fad-oriented. When people say 'research shows,' they generally won't even be able to cite a paper... But if they do and you actually read it, you'll find that it's just unmitigated opinion."
-- David Klein, Ph.D., professor of mathematics, California State University at Northridge

"There are big schools of reading methods and mathematics methods, and so forth, but if you notice, you'll see the reading scores keep going down -- or hardly going up -- in spite of the fact that we continually use these same people to improve the methods. There's a witch doctor remedy that doesn't work. It ought to be looked into; how do they know that their method should work?"
-- Richard Feynman, Nobel Prize laureate

"... ordinary people with commonsense ideas are intimidated by this pseudoscience. A teacher who has some good idea of how to teach her children to read is forced by the school system to do it some other way -- or is even fooled by the school system into thinking that her method is not necessarily a good one."
-- Richard Feynman, Nobel Prize laureate

"It's queer how ready people always are with advice in any real or imaginary emergency, and no matter how many times experience has shown them to be wrong, they continue to set forth their opinions, as if they had received them from the Almighty!"
-- Anne Sullivan

"Education Research: This is a process whereby serious educators discover knowledge that is well known to everybody, and has been for several centuries. Its principal characteristic is that no one pays any attention to it."
-- Neil Postman and Charles Weingartner, The School Book, 1973

"If a physician prescribes a drug that research shows not to work, what happens to the physician? When educators use 'approaches' that research shows not to work, does anything happen -- except to the children who are damaged?"
-- Prof. Martin Kozloff

"As a result, we are no longer to be children, tossed here and there by waves and carried about by every wind of doctrine..."
-- Ephesians 4:14

"Among the community of people who have backing in scientific methods and use objective principles, [we see] a real difference in standards [between] education research that appears in education journals and what appears in psychology and science journals."
-- Louisa Moats, National Institute of Child Health and Human Development

"University schools of education are held in low regard by others in the arts and sciences."
-- Louisa Moats, National Institute of Child Health and Human Development

"Teacher licensing programs are cash cows. ... The schools are interested in filling slots instead of trying to raise low standards. The departments are often segregated and taught by professors of education who've gone through the same (unscientific) program."
-- Louisa Moats, National Institute of Child Health and Human Development

"[Education] students don't get the same grounding in psychology, linguistics, language development or neuropsychology."
-- Louisa Moats, National Institute of Child Health and Human Development

"Whole language made its way into schools with no strong research backing. In fact, the tenets of whole language were actively contradicted by reading psychologists for 25 years. It's appalling."
-- Louisa Moats, National Institute of Child Health and Human Development

"There is no relation between what is practiced in a classroom and what is validated. It is unusual for consumers in education to decide what they are going to do and what materials they are going to buy based on independent, scientific validation."
-- Louisa Moats, National Institute of Child Health and Human Development

"The plural of 'anecdote' is not 'data'."
-- attributed to a variety of writers



Also see the full page on education quotations.