[NLA] NLA Discussion: Evidence-based education

[Catherine B. King]

Hello John:

I want to be as brief as possible in answering your
well-thought out note on evidenced-based education,
and so I may sound curt--but please know I don't
mean to be.   To be brief:  you missed the point and
the point is already pointing to the political.

First:  I MORE than agree that scientific method and
its general applications of evidence collecting--as
you have so generously noted--should not be changed.

The problem is not the method, but rather the data:
problems and differences are the development of the
data **prior to and after** the method is applied.  Though
you have hinted at this movement in your note, I doubt
the problem has been understood at its foundational
level.   With human beings, these questions matter,
and up to this point, we have been unwilling to "go
there" for our fear of losing scientific method if we do--
wrong.

With human data, scientists must be open to critique at
both ends of the scientific cycle--at the beginning where
the questions are developed (hermeneutics) and at the
end where the ethical and political implications come
into play (epistemology and metaphysics).

Up to this point in time, and over the rise of the scientific
method, we have failed to distinguish scientific method--
both classical and statistical--from its data and the
surrounding approaches--the "hard" non-conscious data
that it was first rightly applied to.

And many scientists in the human fields, including statistics
and philosophical theory, have tried to abstract themselves
and these self-critical questions from both ends of the cycle (the
questions at the beginning and the ethical-political ground
at the end) as if, in order to be "scientific," we need to
come at human data as natural scientists come at their
data--again, wrong.

Consequently, many in the field have been guided by the
wrong assumption that if we challenge the above scene
on any ground, then are de facto threatening the whole
scientific project and all of its hard won gains--also wrong.

The failure is not in trying to disregard scientific method,
but rather in failing to distinguish the different forms of
data, data collection, the import of the prior development
of the scientists' questions on humans and their outcomes,
and the ethical-political implications of the outcomes
themselves, including the covert assumption that
complete predictability is the goal--a dangerous
political assumption itself when considering the
open-ended creativity that lies at the basis of all human
education and development  (this assumption has always
been just one of the problems of positivism--a long-time
subject of conversations here).

There is also the scandalous point that the question of
the end result may, in some cases, be legitimately involved
in the venue of the data. That is, unlike "hard" data, adults
may want to be involved in their own hypotheses.

And also consequently, those with a political agenda that
differs from many AE educators are first to use this specious
argument:  "you are killing scientific method if you ask those
questions,"  and "we are only involved with liberal whims," etc.,
(see George's quote from DOE documents and Gail's
introductory paragraph to her "Making the Case"
collection).

I have spoken here before (if you read my notes) about the
development of new paradigms of research that are named
and that have begun to develop critical science from both
ends:  from a critique of the prior development of the
scientists' questions and from the question of outcome,
predictability, and their political implications.  It is naive to NOT
take these things into account in human sciences.  But as
long as we confuse this valid critique with an attack on scientific
method as "evidence based," our field will never progress.

Your note suggests the goal is to leave these questions out
altogether, and this is part of the problem, even in the natural
sciences--there is no view that is a-ethical or a-political.   This
is the hard lesson that technology is finally teaching our
natural scientists.

The point is:  It's not scientific at all to NOT bring critique to
these two domains--the beginning and the end--because the
scientists themselves live in both these domains.  There is
no scientific view that is a-ethical or a-political.  The point is
to know what those are, not to pretend that they don't exist
in the covert interest in yours, or because your research
happens to go along with the status quo.

But it seems that it behooves some not to change with
new understanding, and to claim that their own use of scientific
method in human fields--as developed under one and
only one paradigm of research--either (1) has no ethical
or political implications, or (2) that it doesn't matter because
the way they form their questions and the ethical-political
implications that flow from their own hegemonic
development of the questions and expectations of outcomes
is the ONLY RIGHT and SCIENTIFIC WAY.  If we leave it like
it is, this view will win the day.   I don't know that you are
looking close enough at it.

That "day" has vast political implications, three of  which
are:

(1) a continued arrogance that has dominated the human
scientific fields for years--that "scientists" with these political
views know better and are thus supported by a "camp" of
political agendas who have a great stake in keeping poor
adults ignorant--throwing good money after bad.

(2) that adult "data" have no voice or say about what they want
or no rights to an education unless the "evidence" shows that
our programs "work;"--FIRST--and despite the fact that all
of us claim to embrace a democratic tradition with documents
that state and infer otherwise.   (Your note shows the "decreasing
returns" element of research as it becomes more and more
remote from the current idea of "proof."   The point is, under
the present paradigm, the current political powers are losing
patience with us?)

(3)  that education is not about the yet unknown future or
driven by each and every person in a democratic political
domain, that "success" is not personal and individual, and
not defined partly by us, and that the future must be known
now by the scientists and test-givers--as absolutely
predictable--and we must have those predictions fulfilled--
in order to legitimate funding for programs.  (This trend is
connected at its center to the testing mania in K-12.)

Though you claim an openness to dialogue, and though
the last part of your note was heartening,  I must wonder
if:

(1) the openness you imply is being embraced and
implemented by those who are making decisions, especially
where adult education is concerned--and its continuing
need for research money, as you suggest;

(2) whether the voices of adults and educators in the field
are ever really lent any but pathetic token ears;

(3) whether the need for research in AE you embrace is
losing the small political rug it is already standing on partly
and precisely because of a misapprehension of the "scientific
evidence" argument; and

(4) whether having a plethora of research "available" will
soon turn into having a plethora of **mandated** applications
that **must** be applied thusly or else--fill in the blank--
subverting the crucial, but unenforceable, notion that teachers
are, as you say, the ultimate artists who chose and implement
good theory according to need.

Thank you for listening,

Catherine King
Adjunct Instructor
Department of Education
National University
San Diego, CA

----- Original Message -----
From: John Comings <comingjo at gse.harvard.edu>
To: <nla at lists.literacytent.org>
Sent: Wednesday, May 01, 2002 1:40 PM
Subject: [NLA] NLA Discussion: Evidence-based education


> I missed the "evidence-based education" discussion when it took place but
> wanted to add my two cents, which turned out to be a bit long:
>
> "Evidence-based education" is a complicated term, and I'm worried that the
> political nature of the term (that is, the term is being used to support a
> political position) will lead us away from considering its value.
> Evidence-based education provides us with a way to make decisions about
> policy and practice that can be more effective than the ways we are making
> those decisions now.
>
> While engaged in this debate, we should be clear about how evidence-based
> education is defined.  The scientific method, which is the foundation of
> evidence-based education, doesn't necessarily privilege some research
> methods over others, but it does identify different roles for methods that
> depend on the goal and context of a specific study.  Each method has rules
> as to how data should be collected and analyzed.  Evidence-based education
> is not a single method employed in a one-time test; it is a series of
> stages of inquiry that continue over a long period of time.
>
> In the first stage, a wide range of methods, both qualitative and
> quantitative, are used to explore instruction and develop a hypothesis
> about how to improve it. The evidence that informs the development of the
> hypothesis can come from many different disciplines. This is the
> exploratory stage of the process.
>
> The hypothesis (an approach to instruction) is then tested with
> experimental or quasi-experimental methods (random assignment or
> control-comparison groups with statistical controls) to see if one
approach
> works better than another. This is the confirmatory stage, but research
> never provides a 100% confirmation, just a probability. That is,
> instructional approach "A" works well with 40% of students and approach
"B"
> works well with 60% of students.
>
> Then, the findings of the experiment are explored with a wide range of
> methods to find out why the approach worked with some students and not
> others -- another exploratory stage -- and research from other disciplines
> might point the way to a more effective approach. This second exploratory
> stage leads to design of a new instructional approach that might work well
> with a larger proportion of students or to the addition of a second
> approach for the 40% not served well by approach "B." Then, another
> confirmatory stage takes place to test the new hypothesis.
>
> Approaches that have been proven to work but that have been superceded by
a
> new approach are not abandoned completely.  Elements of them survive, and
> teachers use these "less effective" approaches because they work well with
> some students.  This process continues exploring and confirming until
> almost all students are served well, not with a single approach but with a
> range of approaches that have been proven effective.
>
> As this process continues, research may find that there is a limit to a
> particular line of enquiry.  Approaches A, B, C, etc. might never be
> successful with more than 60% of students. Research then needs a new
> paradigm that might say, for example, that the instructional approach is
> not the problem.  The barrier to success may be the effects of poverty,
> racism, or nutrition; the need for incentives, support services, or
> counseling; or untrained teachers who are implementing the approach
> incorrectly. Research explores in these directions and proposes another
> experiment.
>
> The whole process only works when there is a consensus about the outcomes
> of education because an experiment needs a well-defined outcome measure.
> Some outcomes, reading comprehension or oral vocabulary for example, are
> easily measured by a test. Other outcomes, changes in reading behaviors or
> the use of English at work for example, are difficult, but can be measured
> by observations and interviews.  For some outcomes, increased critical
> thinking ability or improvement in self-efficacy for example, measurement
> may be extremely difficult and require a complicated expensive measurement
> tool, which would be appropriate for research but not for an
accountability
> system.  Some outcomes, enhancement of democracy in the wider society, may
> be impossible to measure, but these outcomes can sometimes be broken down
> into parts that can be measured, such as voting behavior, participation in
> advocacy efforts, or knowledge about political issues.
>
> The experiment stage of the scientific method is difficult in educational
> settings and is particularly difficult in our field.  It can also lead to
> advice that works in the experiment but doesn't work in real programs.
So,
> once an approach to education is shown to work in a small, controlled
> experiment, it must be tested in real programs on a larger scale.  This
> makes experiments very expensive, and so they must be carefully planned.
>
> When research produces findings, the science ends and the art of teaching
> begins. The development of "teacher knowledge" is part of evidence-based
> education.  After research suggests what should be done, teachers develop
> ways to do it that work best for their students.  The development of
> teacher knowledge should be supported as part of evidence-based education,
> and it should be systematically recorded and shared.
>
> Evidence-based education offers the possibility to build, over time, a
body
> of accepted practice through a consensus on how to make instructional
> decisions rather than through a consensus on a philosophy of education.
Of
> course, any system of decision making can be misused, and so if our field
> moves (or is pushed) toward evidence-based education, we should demand a
> clear but complex definition of what it is.
>
> If our field is going to make decisions about policy and practice using
> evidence-based education, we need a lot of new research funded over a long
> period of time and a system for putting that research into practice. That
> research has to be protected from political forces (both from the left and
> the right) that want it to focus on a narrow set of instructional
> approaches.  Practitioners and policy makers have to be trained to read
> research so that they can make decisions when the same research is being
> used to support two different approaches.
>The development of
> teacher knowledge should be supported as part of evidence-based education,
> and it should be systematically recorded and shared.

> Evidence-based education will require support to research that is
> sufficient, in terms of funding and duration, and that encourages
> interaction and cooperation among researchers. Research should follow a
> plan built through a dialogue that includes the voices of researchers,
> policy makers, practitioners, and students.  And, our field will need a
> national system, connected to state professional development systems, that
> makes research available to policy makers and practitioners and shares
> teacher knowledge derived through putting research into practice.
>
> The definition of evidence-based education that I have heard from the
> leadership in the US Dept. of Education's Office of Educational Research
> and Improvement includes separation from political influence, acceptance
of
> a range of methodologies in the exploratory phase, and inclusion of
teacher
> knowledge.  Unfortunately, I don't see our field's piece of the research
> pie clearly identified in the draft legislation for the reform of OERI,
nor
> do I see sufficient funding for linking research to practice or
> implementing evidence-based practice.  I feel we should accept
> evidence-based education (while defending it against inappropriate use as
a
> political tool) and fight for our piece of the pie.  When the Department
of
> Education announces a new initiative for our field, we should ask to see
> the evidence that the initiative is sound and an improvement on existing
> practice.
>
> John Comings, NCSALL
>
> John Comings, Director                    john_comings at harvard.edu
> NCSALL -- Nichols House                   http://ncsall.gse.harvard.edu
> Harvard Graduate School of Education      (617) 496-0516, voice
> 7 Appian Way                              (617) 335-9839, cell
> Cambridge MA 02138                        (617) 495-4811, fax
>
>
> _______________________________________________
> NLA mailing list: NLA at lists.literacytent.org
> http://lists.literacytent.org/mailman/listinfo/nla
> LiteracyTent: web hosting, news, community and goodies for literacy
> http://literacytent.org

_______________________________________________
NLA mailing list: NLA at lists.literacytent.org
http://lists.literacytent.org/mailman/listinfo/nla
LiteracyTent: web hosting, news, community and goodies for literacy
http://literacytent.org