[NLA] NLA Discussion: Evidence-based education

[John Comings]

I missed the "evidence-based education" discussion when it took place but 
wanted to add my two cents, which turned out to be a bit long:

"Evidence-based education" is a complicated term, and I'm worried that the 
political nature of the term (that is, the term is being used to support a 
political position) will lead us away from considering its value. 
Evidence-based education provides us with a way to make decisions about 
policy and practice that can be more effective than the ways we are making 
those decisions now.

While engaged in this debate, we should be clear about how evidence-based 
education is defined.  The scientific method, which is the foundation of 
evidence-based education, doesn't necessarily privilege some research 
methods over others, but it does identify different roles for methods that 
depend on the goal and context of a specific study.  Each method has rules 
as to how data should be collected and analyzed.  Evidence-based education 
is not a single method employed in a one-time test; it is a series of 
stages of inquiry that continue over a long period of time.

In the first stage, a wide range of methods, both qualitative and 
quantitative, are used to explore instruction and develop a hypothesis 
about how to improve it. The evidence that informs the development of the 
hypothesis can come from many different disciplines. This is the 
exploratory stage of the process.

The hypothesis (an approach to instruction) is then tested with 
experimental or quasi-experimental methods (random assignment or 
control-comparison groups with statistical controls) to see if one approach 
works better than another. This is the confirmatory stage, but research 
never provides a 100% confirmation, just a probability. That is, 
instructional approach "A" works well with 40% of students and approach "B" 
works well with 60% of students.

Then, the findings of the experiment are explored with a wide range of 
methods to find out why the approach worked with some students and not 
others -- another exploratory stage -- and research from other disciplines 
might point the way to a more effective approach. This second exploratory 
stage leads to design of a new instructional approach that might work well 
with a larger proportion of students or to the addition of a second 
approach for the 40% not served well by approach "B." Then, another 
confirmatory stage takes place to test the new hypothesis.

Approaches that have been proven to work but that have been superceded by a 
new approach are not abandoned completely.  Elements of them survive, and 
teachers use these "less effective" approaches because they work well with 
some students.  This process continues exploring and confirming until 
almost all students are served well, not with a single approach but with a 
range of approaches that have been proven effective.

As this process continues, research may find that there is a limit to a 
particular line of enquiry.  Approaches A, B, C, etc. might never be 
successful with more than 60% of students. Research then needs a new 
paradigm that might say, for example, that the instructional approach is 
not the problem.  The barrier to success may be the effects of poverty, 
racism, or nutrition; the need for incentives, support services, or 
counseling; or untrained teachers who are implementing the approach 
incorrectly. Research explores in these directions and proposes another 
experiment.

The whole process only works when there is a consensus about the outcomes 
of education because an experiment needs a well-defined outcome measure. 
Some outcomes, reading comprehension or oral vocabulary for example, are 
easily measured by a test. Other outcomes, changes in reading behaviors or 
the use of English at work for example, are difficult, but can be measured 
by observations and interviews.  For some outcomes, increased critical 
thinking ability or improvement in self-efficacy for example, measurement 
may be extremely difficult and require a complicated expensive measurement 
tool, which would be appropriate for research but not for an accountability 
system.  Some outcomes, enhancement of democracy in the wider society, may 
be impossible to measure, but these outcomes can sometimes be broken down 
into parts that can be measured, such as voting behavior, participation in 
advocacy efforts, or knowledge about political issues.

The experiment stage of the scientific method is difficult in educational 
settings and is particularly difficult in our field.  It can also lead to 
advice that works in the experiment but doesn't work in real programs.  So, 
once an approach to education is shown to work in a small, controlled 
experiment, it must be tested in real programs on a larger scale.  This 
makes experiments very expensive, and so they must be carefully planned.

When research produces findings, the science ends and the art of teaching 
begins. The development of "teacher knowledge" is part of evidence-based 
education.  After research suggests what should be done, teachers develop 
ways to do it that work best for their students.  The development of 
teacher knowledge should be supported as part of evidence-based education, 
and it should be systematically recorded and shared.

Evidence-based education offers the possibility to build, over time, a body 
of accepted practice through a consensus on how to make instructional 
decisions rather than through a consensus on a philosophy of education.  Of 
course, any system of decision making can be misused, and so if our field 
moves (or is pushed) toward evidence-based education, we should demand a 
clear but complex definition of what it is.

If our field is going to make decisions about policy and practice using 
evidence-based education, we need a lot of new research funded over a long 
period of time and a system for putting that research into practice. That 
research has to be protected from political forces (both from the left and 
the right) that want it to focus on a narrow set of instructional 
approaches.  Practitioners and policy makers have to be trained to read 
research so that they can make decisions when the same research is being 
used to support two different approaches.

Evidence-based education will require support to research that is 
sufficient, in terms of funding and duration, and that encourages 
interaction and cooperation among researchers. Research should follow a 
plan built through a dialogue that includes the voices of researchers, 
policy makers, practitioners, and students.  And, our field will need a 
national system, connected to state professional development systems, that 
makes research available to policy makers and practitioners and shares 
teacher knowledge derived through putting research into practice.

The definition of evidence-based education that I have heard from the 
leadership in the US Dept. of Education's Office of Educational Research 
and Improvement includes separation from political influence, acceptance of 
a range of methodologies in the exploratory phase, and inclusion of teacher 
knowledge.  Unfortunately, I don't see our field's piece of the research 
pie clearly identified in the draft legislation for the reform of OERI, nor 
do I see sufficient funding for linking research to practice or 
implementing evidence-based practice.  I feel we should accept 
evidence-based education (while defending it against inappropriate use as a 
political tool) and fight for our piece of the pie.  When the Department of 
Education announces a new initiative for our field, we should ask to see 
the evidence that the initiative is sound and an improvement on existing 
practice.

John Comings, NCSALL

John Comings, Director                    john_comings at harvard.edu
NCSALL -- Nichols House                   http://ncsall.gse.harvard.edu
Harvard Graduate School of Education      (617) 496-0516, voice
7 Appian Way                              (617) 335-9839, cell
Cambridge MA 02138                        (617) 495-4811, fax


_______________________________________________
NLA mailing list: NLA at lists.literacytent.org
http://lists.literacytent.org/mailman/listinfo/nla
LiteracyTent: web hosting, news, community and goodies for literacy
http://literacytent.org