[NLA] Can research improve policy or practice?

[Bickerton, Robert P]

 Tom, et al,

Thanks for raising this issue.  A few thoughts from one of several state ABE
directors who tried to influence the direction of that $50 million back in
1993 (or thereabouts).

By (about) 1993, the R&D initiatives at USDOE (DAEL/PAS and OERI) and NIFL
were starting to gell.  We were concerned that what appeared to be emerging
were a series of small R&D efforts that were neither strategic nor tied
together in a coherent thematic way.  In our discussions with the principal
players, we asked that the three major players (DAEL National Programs,
NIFL, and NCAL -- the predecessor to NCSALL) work together on a strategic
and staged approach to R&D.  In particular, many of us argued them to tackle
assessment.  We communicated the high level of dissatisfaction practitioners
had with existing commercially available assessments, that many of us shared
their dissatisfaction, and that the challenge was beyond the capability
(read, "financial resources") of many states to tackle alone.  

Our entreaties in this regard were largely ignored.  It is true that the
state directors and others have been regularly consulted on which National
Programs and NIFL projects deserved support, but the choices have been among
several disconnected smaller initiatives that many on this list are aware
of.  There has not been a large scale coordinated effort to address the
assessment issue which is now at the heart of what is attempting to serve as
an ABE accountabilty system.  Will anyone argue that by the 1999/2000 and
2000/2001 program years that the states were prepared to provide valid,
reliable and comparable data about the educational progress students are
making?  And will anyone argue that we will do a significantly better job
when we report our program year 2001/2002 results?

A case in point that illustrates why the failure to tackle this difficult
systemic issue is so problematic.  There are two multi-year research studies
being conducted by USDOE with National Programs funding:  "What Works in
ABE" and "What Works in ES(O)L."  Each was originally funded at $1 million
per year for five years -- a total of $10 million over 5 years.  With these
studies it's important to keep in mind that without objective measures of
whether students are learning and succeeding, the answer to "what works" is
nothing more than who markets programs and services better.  In fact, the
ABE study tried to use uniformed interns to make judgements about which
programs were most likely to harbor exemplary practices after 1 or 2 visits
to a random set of programs.  [NOTE:  I and a couple of other people raised
such hell about the impending waste of scarce resources that the study was
taken off line for several months and a new contractor hired.] In the
absence of solid tools that practitioners could agree upon, both studies
spent most of the first 18 months struggling over which assessments to
"settle on," or in my words, "grit our teeth and make the best of a bad
situation."  

Equipped For the Future is a bright spot in this otherwise dismal landscape.
By starting with "what do adults want/need to know and be able to do," EFF
has tackled the real first step in developing and achieving consensus around
adult ed assessments.  For assessments to be valid and reliable, they must
align with what's being taught, i.e., curriculum.  Absent agreement about
the "universe of content," we can neither select nor construct meaningful
assessments.  Thank you for a good first step.  

But the challenge remains.  EFF is working on the assessment challenge but
they're not there yet.  Further, EFF appear's to be restricting its work to
performance assessment only.  While many of us agree that performance
asssessment can be the most promising and meaningful approach to student
assessment in adult ed, it is highly unlikely that all the challenges in
achieving validity, reliability and comparability can be resolved in a way
that is affordable in both time and money for our programs.  Some of us are
concluding that at least for the present, we will need a mix of performance
and paper and pencil assessments.  No such options appear to be emerging
nationally.

So what does this do to a state like Massachusetts?  We have convened a
"performance accountability working group" of very diverse and committed
practitioners and experts to help resolve the challenges presented by
performance accountability -- with a particular emphasis on assessment of
learning gains.  We have watched about $50 million expended on adult ed R&D
and find little to nothing of use to this effort. We are struggling with the
reality that our state may need to divert a million dollars or more away
from instruction to do the development that our leaders were asked to
undertake almost a decade ago.  We're not happy about this.

I believe our national leadership understands that the funds do not belong
to them but to our field.  I believe they also seek our feedback about what
to do with these funds -- and they have.  The problem is that we speak with
so many voices that what they end up with is long "wish lists" and the
result is a watered down set of activities intended to do something for as
many items on the list as possible.  We own part of the problem Tom, I and
others are highlighting.  Our goal must NOT be to simply have input. Our
goal must be to thrash through our many different interests, identify common
ground, and speak with a single voice about the order in which our top
priorities can and should be addressed.  We rarely do this.  On those rare
occassion when we have, we've prevailed.  

Do you want to see the next $50 million spent in a way that builds a strong
foundation for our field?  Then be prepared for some hard work -- not just
in getting national leaders to listen, but to arrive at that common ground
among your colleagues.  

bob bickerton, MA director of adult ed



-----Original Message-----
From: Thomas Sticht
To: nla at lists.literacytent.org
Cc: tsticht at aznet.net
Sent: 11/8/01 11:53 AM
Subject: [NLA] Can research improve policy or practice?

Research note							November
8, 2001

Tom Sticht

Can adult literacy research improve policy or practice?

I recently attended a couple of R & D planning meetings which got me
wondering if the 50 million or so dollars that the federally-funded
organizations that have responsibility for research (National Center for
Adult Literacy, National Center for Adult Learning and Literacy,
National Institute for Literacy, Division for Adult Education and
Literacy) have spent in the last 10-12 years have made much improvement
in either policy or practice. 

While all these organizations have been pursuing research and publishing
reports,  non-addressed issues have arisen that deal with major
consequences for the adult literacy education field. 

Scale of Need. Important  questions of the scale of need for adult
literacy education have arisen, with the National Center for Education
Statistics producing a report by the former director of the National
Adult Literacy Survey (NALS) saying that the NALS used the wrong methods
and that the scale of need is only half of what was earlier said.
Despite this, R & D organizations go on using the NALS data as though
they are valid indicators of adult's literacy skills in the USA and
elsewhere. There is also a huge gap between the numbers of adults
sampled on the NALS who thought they had a reading problem, some 95
percent thought they did not, and the reports of literacy problems
derived from the NALS standardized tests. This would seem to pose a
major problem in recruitment and increasing participation in adult
literacy education. If people don't think they have a literacy problem,
why should they seek literacy education as a solution? We need some
solid research about how many adults need basic skills instruction and
how many want it. I haven't found any research institute expressing
concern about this or doing anything to educate the field about all
this. But perhaps I am just not aware of what R & D centers are doing
about this and someone will inform me. 

Access and Participation. U. S. Department of Education officials report
that enrollments in the Adult Education and Literacy System of the
United States plummeted from over 4 million in 1997 to around 2.9
million in 2000. This is a drop of over 25 percent,  but I haven't found
any research institutes expressing concern about this, even though all
of the federally funded R & D organizations  (NCAL;NCSALL; NIFL;DAEL)
have had participation and retention as one of their major interests. 

Nature of Provision.  To my knowledge no R & D center has stated concern
about the proliferation of non-validated ideas about literacy practice
and "reform" expressed in reports from  government funded organizations,
including the R & D centers themselves. A variety of reports from these
organizations tell teachers that they and their students can benefit
from the new knowledge contained in the reports. Yet they offer very
little, if any, concrete, convincing  evidence that some
teaching/learning problems are solved or improved by anyone who
possesses the knowledge given in the numerous reports.  

Accountability.  Numerous issues regarding the National Reporting System
and its encouragement of the use of standardized tests that almost all
in the R & D field acknowledge are inadequate measures of adult learning
in adult literacy education programs have been raised in various
refereed journals, literacy newsletters,   and internet lists, including
the NLA  list.  Yet  R & D institutes have had hardly a word to say
about this, though NCSALL published a report by Juliet Merrifield that
raised a number of issues that I have not seen addressed yet by NCSALL
or any other federal adult research organization. But perhaps the
National Academy of Science committee formed recently by DAEL will
address many of these issues. 

Another aspect of accountability is the need for data regarding the
returns to investments in adult education and literacy development in
many areas. I give numerous speeches advocating for adult literacy
education in different nations each year and I repeatedly hear about the
need for ROI information.  Can adult education produce better health
care for adults and their children and produce savings in medical costs?
Does it produce savings in early childhood, compensatory education
costs, does it produce savings in training costs in business and
industry? Does it produce returns in increased productivity at work? It
is very difficult to find "hard" evidence to argue for support for the
AELS, yet there does not seem to be much such evidence being sought in
the R & D centers for adult education and literacy research that would
permit strong ROI arguments for advocating for increasing funds for
adult education and literacy development. 

Finally, in the meetings to plan future research agendas for adult
literacy and family literacy education that I recently attended I did
not hear much to lead me to suppose that the next decade will bring much
more by way of R & D that can actually improve policy or practice. For
one thing, there was no actual policy or practice problems or issues
that were identified as existing in some real place or in some real
policy that needed research to inform its change. Instead of considering
real problems of policy or practice in some real contexts, the R & D was
placed in a decontextualized frame and discussed not as problems but
rather as academic topics that need to be researched to "fill in the
gaps" in "our" knowledge, as though there are such metaphorical "gaps"
which "we" can just "plug" with a little R & D.  

The advantage of the topic approach to R & D is that it permits endless
theses and dissertations by graduate students in academic institutions
because as is well known, there is no end to the exploration of a topic.
Unfortunately, in my experience, there is also a very small track record
for such topic-oriented  R & D to solve genuine practical problems in
adult literacy education in some real place and real time (or any other
problems of education, witness the billions of dollars spent on K-12 R &
D over the last quarter century and the present state of education). 

Over the years I have been a strong advocate for R & D in adult literacy
education. But right now, the lack of any major responses to the
national problems I've mentioned above, and the lack of any evidence of
effectiveness in solving important adult literacy education problems in
various R & D reports I have recently seen from federally sponsored and
supported research and dissemination organizations make me very
concerned that if things continue in the future as they have in the
past, the field will not find much benefit from another decade of R & D
in adult literacy education.  

But perhaps I am overly pessimistic. Perhaps readers will respond with
numerous insights into how R & D has solved very practical policy and
practice problems for them and they have some data, some convincing
evidence,  to substantiate such claims. I hope so.
_______________________________________________
NLA mailing list: NLA at lists.literacytent.org
http://lists.literacytent.org/mailman/listinfo/nla
LiteracyTent: web hosting, news, community and goodies for literacy
http://literacytent.org