[AAACE-NLA] Getting Hype Not Help

[Bickerton, Robert P]

George, et al,

First, I hope this note finds you and your program doing well.

Second, I have another perspective to add to the debate on "scientifically
based research."  It comes from the years I spent as policy/legislative
chair for both AAACE and the State Directors of Adult Ed.

Most of the postings I have read are reacting to the context within which
this policy is being promoted, i.e., the degree to which a practice must be
scientifically based is directly propotional to how in favor or not in favor
it may be with certain "leadership" positions.  For example, Even Start is
being eviscerated allegedly because of a lack of definitive impact data yet
funding for faith based initiatives is climbing astronomically.  Please
note, that while I am highly critical of the proposed cuts to Even Start I
am not arguing against funding for faith based initiatives.  We fund many of
the latter in ABE in Massachusetts although we have always done so with a
clear separation between the services and whatever faith may be sponsoring
them.  What I am highlighting is the inherent contradiction and that these
contradictions appear to be widespread depending on the philosophy of the
proponents.  So the skepticism so many have expressed on this listserv is
well founded.

On the other hand, I believe it is a mistake to so substantially dismiss
what we could be gaining -- in fact, need to be gaining from more systematic
and "riorous" research.  Toward this end, I submit the following two
situations for your consideration.  

1.  MRDC is highly regarded by many federal policy leaders including members
of Congress and key staffers.  I have been troubled by the findings and
recommendations that have come from some of their research and encouraged by
others.  I imagine most of you know the research project they did in
Riverside, CA quite awhile ago (15 years ago or so?).  It found that
education and training made no statistically significant difference in the
earnings potential of adults on public assistance.  It was used over and
over again to support the "work first" policies that were then emerging as
part of "welfare reform."  Everyone was quoting this study.  I read the
study as well as all related research design materials.  It was, in fact,
relatively well designed -- probably as good as you can get in an education
and human service setting given the constraints / confounding dynamics that
so many of you have pointed out.  My problem with it was that a single
program's "treatment approach" was used to generalize to ALL adult education
and training.  With a background in Physics (my first major), I found and
continue to find this absurd.  We can poke holes in policy language
surrounding "replicability" but quite frankly, that is how that study needed
to be challenged.  A whole generation of very poor, primarily single parent
(mostly female headed) households have been hurt by the policies that this
study helped to prop up.  Rather than attempt to hold our fingers in the
dyke (or be "saboteurs" of this trend -- i.e., throwing our "sabots"/shoes
into the machinery of data driven policy making), we should be holding
policy leaders accountable to the same standards they are promoting.

2. I've been to ABE/literacy/ESOL/workplace/family literacy meetings in most
states across our nation.  Everywhere one goes, you hear assertions that
this program or that practice is extraordinarily effective.  These
assertions are always accompanied by the claim that each deserves more
funding.  Now, each of the subscribers to this list will look at some of
these practices with great appreciation for what their colleagues are doing.
In other cases, you may be horrified.  Perhaps we all believe that we'd do a
better job as "judge and jury" for who and what deserves the designation of
"effective" and that should receive abundant funding.  Well, most people who
have become elected and policy leaders feel the same way about themselves.
Regardless, I'm almost as uncomfortable with the notion that some collection
of adult educators would apportion funding based on their "experience" and
instincts as I am with an allegedly data driven process.  Before you blow a
gasket, the reason I'd be uncomfortable with such an approach is because
I've watched the following dynamic -- those who are best at self-promotion
tend to land such privaleged perches.  I hope you'll agree that the
correlation between effective/"good" ABE/literacy/ESOL services and
self-promotion can be quite limited.  

I believe the drive to better information / "data" will ultimately help our
field -- but only IF:  
(a) practitioners and students are an integral part of the design, analysis
and policy processes;
(b) quantitative data is balanced by qualitative information based to the
maximum feasible extent on expert opinion and judgement -- academic AND as
our European colleagues say, "on the ground."
(c) the data is accurately reflects "real"/authentic/meaningful measures
that are grounded in the collective wisdom of all relevant constituences.  
(d) the measures are appropriate at ALL levels of proficiency so that any
tendency to "cream" (to serve only those at higher levels and who would
probably be successful without instruction) is "frustrated"/eliminated.

Accomplishing these criteria is really difficult.  I've heard many (far too
many) practitioners in our field lament their limited power/influence over
every aspect of these issues and the processes by which they are resolved.
I believe this has contributed to so many holding diametrically opposite
positions to the policies and practices that inform federal and state
actions.  (By the way, this is NOT how I read George's piece which I read as
a thoughtful critique that sought out both the pros and the cons.) I
maintain that this "rejectionist" approach is neither a useful dynamic nor a
strategy destined for success.  You have much more power than you give
yourself credit for.  You do not have the power to stop the forces driving
all of education and human services toward data driven decision making --
for every fault you can find there is an offsetting benefit others will
point to.  (The health field figured this out almost a century ago!)  What
you can do -- and, I believe, must do -- is to engage in the process and
make sure the shape it finally takes works better for our students -- and
for practitioners -- than it will without you.  This does not have to be
"selling out" -- it can, should and must be MAKING A DIFFERENCE -- that you
owe to millions of under-educated and limited English proficient adults
(they need and deserve nothing less) -- and that you owe to yourself.  

Sorry I've gone on so long. 
take care,
bob bickerton, MA director of adult ed and chair of the NCSDAE and NAEDPC
(state directors of adult ed policy and professional development
organizations)


-----Original Message-----
From: George demetrion
To: National Literacy Advocacy List sponsored by AAACE
Sent: 9/24/2004 5:35 PM
Subject: Re: [AAACE-NLA] Getting Hype Not Help

The following is from the What Works Clearinghouse ( http://w-w-c.org
<http://w-w-c.org> ), cited in Tom's message:
 

What is scientifically based research? 



According to the Institute of Education Sciences
<http://www.ed.gov/about/offices/list/ies/index.html> , scientifically
based research: 


*	employs systematic, empirical methods that draw on observation
or experiment; involves data analyses that are adequate to support the
general findings; relies on measurements or observational methods that
provide reliable data; makes claims of causal relationships only in
random-assignment experiments or other designs (to the extent such
designs substantially eliminate plausible competing explanations for the
obtained results); 

I find that to be a reasonable statement on the whole.  Note the
qualifier in the parenthesis, especially the word "eliminate."  That's a
very high threshold and well may beyond what can be substantially proven
particularly when variables cannot be strictly delineated and measured,
which is typically the case for complex human problems relatedto
analyzing educational issues.

*	ensures that studies and methods are presented in sufficient
detail and clarity to allow for replication or, at a minimum, to offer
the opportunity to build systematically on the findings of the research;


Sufficient detail and clarity--to be sure, as a regulative ideal, but
what that means in any given study is up for some interpretation and
perhaps multiple perspectives.  Detail and clarity are qualities of
relative degrees.  The concept of replication denotes an ideal of
exactitude.  The regulative ideals that these terms represent may not be
squarable.  That doesn't mean "anything goes," but neither does it mean
that some exactitude or certainty is necessarily achievable when
speaking of qualities, and, depending on the nature of the study and/or
problem, even desirable.

*	obtains acceptance by a peer-reviewed journal or approval by a
panel of independent experts through a comparably rigorous, objective,
and scientific review; and 

For a formal study, yes, I suppose, on the first half of the statement.
On the second part, I would substitute the word "competent" for
"rigorous," in that the latter term as commonly used has metaphorical
implications in terms of conveying an ideal of exactitude that is not
necessarily accessible to a given study. The term, "competent inquiry,"
used by Dewey and referred to in Philipps & Burbules' Positivism and
Educational Research, may also contain metaphorical implications, but
the term is not as connotative as the term, rigorous.  Moreover,
"competent inquiry" does not necessarily rely on "hard" data (another
metaphor, right?), but on whatever is needed at any stage of an
investigative process, which could include an imaginative hypothesis
that sets the spotlight in a new direction.

*	uses research designs and methods appropriate to the research
question posed. 

Exactly, yet that negates the claim that "experimental design" in its
linkage to random sampling, is the "gold standard" of scientific
research.

George Demetrion

 

(Snippet)

----- Original Message -----
From: Thomas Sticht
Sent: Thursday, September 23, 2004 10:57 PM
To: aaace-nla at lists.literacytent.org
Subject: [AAACE-NLA] Getting Hype Not Help
 
September 23, 2004

Is the Federal Government Practicing Hype
and Not Help in Adult Literacy Education?

Tom Sticht
International Consultant in Adult Education

The What Works Clearinghouse (WWC) (http://w-w-c.org ) was created in
2002
by the U.S. Department of Education's Institute of Education Sciences
(IES) to provide educators, policymakers, researchers, and the public
with
a central and trusted source of scientific evidence of what works in
education. The internet web site of the What Works Clearinghouse boldly
proclaims that it is  "A trusted source of scientific evidence of what
works in education."



 <<ATT39724.txt>>