[NLA] NLA Discussion: Evidence-based education

[Sioux Falls Area Literacy Council]

Dear John & NLA subscribers:

I read with interest your email regarding this important topic.  I also
scanned this morning George Demetrion's email, but am responding to just
points in yours read last night at home.

JOHN WROTE:
> Evidence-based education will require support to research that is
> sufficient, in terms of funding and duration, and that encourages
> interaction and cooperation among researchers. Research should follow a
> plan built through a dialogue that includes the voices of researchers,
> policy makers, practitioners, and students.  And, our field will need a
> national system, connected to state professional development systems, that
> makes research available to policy makers and practitioners and shares
> teacher knowledge derived through putting research into practice.

The above paragraph (which you wrote near the end
of your lengthy e-mail) shows the fallacy of even approaching
"evidence-based
education" if this is the process it will take.

Our field can't even get cooperation and collaboration of half of these
people listed to merely design a satisfactory testing tool that truly
measures the beginning level of a literacy student's ability and then gives
a
valid benchmark of progress thereafter.  If the "field" can't get funding
for
such a much needed accountability *tool*, whatever would lead any of us to
believe that they'd fund such research and "a national system connected to
state professional development systems", as you write here?

What kind of literacy program are you using as an example?  It's obviously
not the volunteer- driven program like most literacy programs I've seen:
Programs administered by up to five paid staff members and instruction given
by volunteers; learners whose desire may be to learn to write the name of
their street!  And in the meantime, we program providers are going to
advocate for a lengthy plan within which the above student, a 40-year old
man, is expected to sit back and wait until the dialogue ends so he can fill
out a form with his correctly spelled address.  Am I correct in this
thinking?

JOHN WROTE:
> Evidence-based education provides us with a way to make decisions about
> policy and practice that can be more effective than the ways we are making
> those decisions now.

I can definitely agree with this sentence.  The sad part is this:  If we are
doing it so that we can be more effective in our decision-making, is it not
tied to funds all the way down the line?  If we are all strapped for money
and short-staffed now, how in the world will be able to carry out such
extensive research if the politics are eliminated?  It's the elected
officials who hold the purse strings, isn't it?

JOHN WROTE:
> Each method has rules
> as to how data should be collected and analyzed.  Evidence-based education
> is not a single method employed in a one-time test; it is a series of
> stages of inquiry that continue over a long period of time.

Then you outlined in great detail all those stages, going on to write:

>> The whole process only works when there is a consensus about . a
well-defined outcome measure.  Some outcomes, reading comprehension or oral
vocabulary . are easily measured by a test. Other(s), .. changes in reading
behaviors or the use of English at work, ...  are difficult but can be
measured by observations and interviews.  ... (I)ncreased critical thinking
ability or improvement in self-efficacy,  measurement may be extremely
difficult and require a complicated expensive measurement tool, ..
appropriate for research but not for an accountability system.. >>

I edited some of the wording out to shorten up this e-mail response, John,
but in essence is such a design of such a complicated "outcome measure" of
all levels of achievement even a realistic and reachable goal?  If all that
you listed were to be documented, it would take a lot more personnel than
any
of the smaller programs could even dream of having for the dissemination of
direct *services* - much less to create a satisfactory accountability
system of which the funders would approve.

JOHN WROTE:
> The experiment stage of the scientific method is difficult in educational
> settings and is particularly difficult in our field.  It can also lead to
> advice that works in the experiment but doesn't work in real programs. So,
> once an approach to education is shown to work in a small, controlled
> experiment, it must be tested in real programs on a larger scale.  This
> makes experiments very expensive, and so they must be carefully planned.

John, I really disapprove of the above statements.  Aren't we attempting to
find the answer so that "real programs" effectively approach "learning to
read"?  Why would we ever be looking at a "scientific method" that would
mean
the "educational setting" and "our field" would even agree to "the
experiment"?  It would be a waste of time if so approached, in my opinion.
The word "dollars" pops into my mind *again* when I read this.  Who
exactly is going to pay for this wonderful scientific experiment?  Will the
benefactors truly be the non-reading or low level literacy student?  Or
would
it in the end benefit only the researchers and the program administrators
tallying their numbers?

The literacy education that I offer adult learners in our small program
leans
toward the learner (with help from their volunteer and I) setting personal
life goals - their "evidence".  The question is asked:  What is it they want
to be able to read, write and spell or which activities could they
accomplish
with improved skills? Which other life skills are they currently designating
to advocates or partners in the household?  Yes, this type of program
involves a "range of methodologies" and a virtually immeasurable standard
oftentimes exists since approaching the educational experience of adult
learners in this way is so individualized.

Here's a story about why *I* feel it's important:

I talked on the phone yesterday with a 75-year old mother whose 40-year-old
son is waiting for a tutor to be assigned.  Yes, he still wants to have a
volunteer, she told me.  In fact, she added, she couldn't wait for him to
begin his studies because she knew a woman who used to come to the hamburger
shop where she worked, she told me.  Her whole personality changed after
being in our program, according to this elderly lady.  She didn't know her
name, but she remembered the woman's smile, her new positive attitude and
nobody knew that it was because she learned to read.  She just "came
downtown" once every week and stopped by the restaurant afterwards.  The
woman finally told this elderly lady she was learning to read.  Now it's
going to be her son's turn Mom felt.

The idea is to give "hope" - hope for a different future where
discouragement
and frustration are lessened in the workplace, in the home and/or with their
families.  Let's pursue establishing policy that will be funded to increase
our outreach so more adults, who need help changing their reading
capabilities, get that opportunity in the *current* generation.  All this
"research" says to me that we will go another 50 to 100 years with
illiteracy
in our society while we "study the most effective methodology."

Sorry, John.  I can't agree that policy be based on this lengthy process.

Nancy Hansen
Sioux Falls Area Literacy Council
sfliteracy at mcleodusa.net
Sioux Falls, SD

----- Original Message -----
From: "John Comings" <comingjo at gse.harvard.edu>
To: <nla at lists.literacytent.org>
Sent: Wednesday, May 01, 2002 3:40 PM
Subject: [NLA] NLA Discussion: Evidence-based education


> I missed the "evidence-based education" discussion when it took place but
> wanted to add my two cents, which turned out to be a bit long:
>
> "Evidence-based education" is a complicated term, and I'm worried that the
> political nature of the term (that is, the term is being used to support a
> political position) will lead us away from considering its value.
> Evidence-based education provides us with a way to make decisions about
> policy and practice that can be more effective than the ways we are making
> those decisions now.
>
> While engaged in this debate, we should be clear about how evidence-based
> education is defined.  The scientific method, which is the foundation of
> evidence-based education, doesn't necessarily privilege some research
> methods over others, but it does identify different roles for methods that
> depend on the goal and context of a specific study.  Each method has rules
> as to how data should be collected and analyzed.  Evidence-based education
> is not a single method employed in a one-time test; it is a series of
> stages of inquiry that continue over a long period of time.
>
> In the first stage, a wide range of methods, both qualitative and
> quantitative, are used to explore instruction and develop a hypothesis
> about how to improve it. The evidence that informs the development of the
> hypothesis can come from many different disciplines. This is the
> exploratory stage of the process.
>
> The hypothesis (an approach to instruction) is then tested with
> experimental or quasi-experimental methods (random assignment or
> control-comparison groups with statistical controls) to see if one
approach
> works better than another. This is the confirmatory stage, but research
> never provides a 100% confirmation, just a probability. That is,
> instructional approach "A" works well with 40% of students and approach
"B"
> works well with 60% of students.
>
> Then, the findings of the experiment are explored with a wide range of
> methods to find out why the approach worked with some students and not
> others -- another exploratory stage -- and research from other disciplines
> might point the way to a more effective approach. This second exploratory
> stage leads to design of a new instructional approach that might work well
> with a larger proportion of students or to the addition of a second
> approach for the 40% not served well by approach "B." Then, another
> confirmatory stage takes place to test the new hypothesis.
>
> Approaches that have been proven to work but that have been superceded by
a
> new approach are not abandoned completely.  Elements of them survive, and
> teachers use these "less effective" approaches because they work well with
> some students.  This process continues exploring and confirming until
> almost all students are served well, not with a single approach but with a
> range of approaches that have been proven effective.
>
> As this process continues, research may find that there is a limit to a
> particular line of enquiry.  Approaches A, B, C, etc. might never be
> successful with more than 60% of students. Research then needs a new
> paradigm that might say, for example, that the instructional approach is
> not the problem.  The barrier to success may be the effects of poverty,
> racism, or nutrition; the need for incentives, support services, or
> counseling; or untrained teachers who are implementing the approach
> incorrectly. Research explores in these directions and proposes another
> experiment.
>
> The whole process only works when there is a consensus about the outcomes
> of education because an experiment needs a well-defined outcome measure.
> Some outcomes, reading comprehension or oral vocabulary for example, are
> easily measured by a test. Other outcomes, changes in reading behaviors or
> the use of English at work for example, are difficult, but can be measured
> by observations and interviews.  For some outcomes, increased critical
> thinking ability or improvement in self-efficacy for example, measurement
> may be extremely difficult and require a complicated expensive measurement
> tool, which would be appropriate for research but not for an
accountability
> system.  Some outcomes, enhancement of democracy in the wider society, may
> be impossible to measure, but these outcomes can sometimes be broken down
> into parts that can be measured, such as voting behavior, participation in
> advocacy efforts, or knowledge about political issues.
>
> The experiment stage of the scientific method is difficult in educational
> settings and is particularly difficult in our field.  It can also lead to
> advice that works in the experiment but doesn't work in real programs.
So,
> once an approach to education is shown to work in a small, controlled
> experiment, it must be tested in real programs on a larger scale.  This
> makes experiments very expensive, and so they must be carefully planned.
>
> When research produces findings, the science ends and the art of teaching
> begins. The development of "teacher knowledge" is part of evidence-based
> education.  After research suggests what should be done, teachers develop
> ways to do it that work best for their students.  The development of
> teacher knowledge should be supported as part of evidence-based education,
> and it should be systematically recorded and shared.
>
> Evidence-based education offers the possibility to build, over time, a
body
> of accepted practice through a consensus on how to make instructional
> decisions rather than through a consensus on a philosophy of education.
Of
> course, any system of decision making can be misused, and so if our field
> moves (or is pushed) toward evidence-based education, we should demand a
> clear but complex definition of what it is.
>
> If our field is going to make decisions about policy and practice using
> evidence-based education, we need a lot of new research funded over a long
> period of time and a system for putting that research into practice. That
> research has to be protected from political forces (both from the left and
> the right) that want it to focus on a narrow set of instructional
> approaches.  Practitioners and policy makers have to be trained to read
> research so that they can make decisions when the same research is being
> used to support two different approaches.
>
> Evidence-based education will require support to research that is
> sufficient, in terms of funding and duration, and that encourages
> interaction and cooperation among researchers. Research should follow a
> plan built through a dialogue that includes the voices of researchers,
> policy makers, practitioners, and students.  And, our field will need a
> national system, connected to state professional development systems, that
> makes research available to policy makers and practitioners and shares
> teacher knowledge derived through putting research into practice.
>
> The definition of evidence-based education that I have heard from the
> leadership in the US Dept. of Education's Office of Educational Research
> and Improvement includes separation from political influence, acceptance
of
> a range of methodologies in the exploratory phase, and inclusion of
teacher
> knowledge.  Unfortunately, I don't see our field's piece of the research
> pie clearly identified in the draft legislation for the reform of OERI,
nor
> do I see sufficient funding for linking research to practice or
> implementing evidence-based practice.  I feel we should accept
> evidence-based education (while defending it against inappropriate use as
a
> political tool) and fight for our piece of the pie.  When the Department
of
> Education announces a new initiative for our field, we should ask to see
> the evidence that the initiative is sound and an improvement on existing
> practice.
>
> John Comings, NCSALL
>
> John Comings, Director                    john_comings at harvard.edu
> NCSALL -- Nichols House                   http://ncsall.gse.harvard.edu
> Harvard Graduate School of Education      (617) 496-0516, voice
> 7 Appian Way                              (617) 335-9839, cell
> Cambridge MA 02138                        (617) 495-4811, fax
>
>
> _______________________________________________
> NLA mailing list: NLA at lists.literacytent.org
> http://lists.literacytent.org/mailman/listinfo/nla
> LiteracyTent: web hosting, news, community and goodies for literacy
> http://literacytent.org
>

_______________________________________________
NLA mailing list: NLA at lists.literacytent.org
http://lists.literacytent.org/mailman/listinfo/nla
LiteracyTent: web hosting, news, community and goodies for literacy
http://literacytent.org