[NLA] Can research improve policy or practice?

[Stein, Sondra]

I guess it is not surprising to anyone that the Director of Equipped for the
Future would believe that accountability for results that matter to
learners, programs, and funders is of critical importance.  I appreciate
Bob's characterization of  EFF as a "bright spot in an otherwise dismal
landscape."  We have been working hard over the past seven years to build
consensus on what results matter, so that our system could in fact begin
aligning instruction and assessment with those results - and so that we
could tell Congress exactly how the skills and knowledge adults gained in
our programs enabled them to "compete in a global economy, exercise the
rights and responsibilities of citizenship, and help their children succeed
in school."
While our focus has shifted over the past two years to building an
assessment framework for the EFF Content Standards, we  see our current work
as still focused on defining the "content" for each standard.  Working with
our field research partners in Maine, Ohio, Oregon, Tennessee, and
Washington  we are constructing a "continuum of performance" for each EFF
standard.  The continuum is composed of "snapshots of performance" of adult
literacy and ESOL students on performance assessment tasks that have been
carefully constructed to focus on the "knowledge base" (including
vocabulary, content, and cognitive and metacognitive strategies) required to
use a given standard to carry out the purpose embodied in that task.
Building (and validating) the continua is part of defining the "universe of
content" ( to use Bob's phrase) for each of the EFF standards.  This is what
provides the specification for what performance looks like at any given
point along the continuum - say the transition points from level to level on
the NRS.  Until we have finished this work, it is not possible to construct
assessments based on the EFF Standards that can be used for Accountability
Purposes.

Once we have finished this work, however, any test maker, any publisher of
assessments, can use those definitions of performance (what we call our
cognitive model of performance) to construct ANY KIND OF assessments aligned
with the EFF Standards.  NIFL does not have the resources to construct such
assessments ourselves.  That's why we have established relationships with
all the publishers of assessments currently used in adult education, so that
they will know what we are doing, and will be ready to incorporate our work
into their own products.  We think that the number of states that have
already adopted EFF as a framework means that there is a market that
publishers will want to serve.  

In the meantime, as Bob said, we are concentrating on performance
assessments for several reasons.  Because these are tools that help teachers
focus on and collect evidence of the kinds of cognitively complex
performance that is called upon by everyday activities, and because we don't
now have any tools that enable us to collect this kind of evidence.  We are
excited to be in partnership with DAEL on the National Academy of Science's
Committee on Quality Measures for Alternative Assessments of Adult Literacy.
DAEL will draw on the work of the Committee to issue guidance to State
Directors of Adult Education who use alternative assessments to measure and
report learning gains through the NRS. NIFL will draw on the Committee's
work to assure that performance assessments developed to measure progress on
EFF Standards are of the highest quality and consistent with quality
standards put forth by DAEL. 

So what's our timeline for this work?
By July 2002 we will have validated the continua for the five EFF Standards
currently in the National Reporting System (Read with Understanding, Convey
Ideas in Writing, Listen Actively, Speak So Others can Understand, Use Math
to Solve Problems and Communicate).  Over the Summer we will conduct an
Assessment Task Institute to develop performance assessment tasks that can
be used in relation to NRS levels.  By the following summer we will have
validated Continua for seven more EFF Standards.  

This means that by Summer 2003 EFF will have developed and validated
definitions of performance for 12 of our standards.  These definitions will
support the development of new assessments that hopefully will take us all
out of the dismal swamp - and a little farther toward our vision of a system
in which teaching and assessment, reporting and accountability are all
aligned with and produce results that matter - to our students and to the
public and private agencies that provide resources for adult literacy
services.  

If you would like more information on the work of the EFF Assessment
Consortium please visit the 4EFF Archives (see messages by Brenda Bell and
Peggy McGuire for April, May, and June 2001) - or feel free to be in touch
with Brenda Bell or Regie Stites, Coordinators of the Consortium. 

And if anyone has a million dollars or so you'd like to give us - the
Consortium would be delighted to speed up and extend our work on assessment
and move on to credentialing.

Sondra


Sondra G. Stein, PhD.
Senior Research Associate and
National Director, Equipped for the Future
National Institute for Literacy
1775 I Street NW   Suite 730
Washington, DC 20006
phone:  202-233-2041
 fax:     202-233-2050

 -----Original Message-----
From: 	Bickerton, Robert P [mailto:RBickerton at doe.mass.edu] 
Sent:	Friday, November 09, 2001 8:20 AM
To:	'Thomas Sticht '; 'nla at lists.literacytent.org '
Subject:	RE: [NLA] Can research improve policy or practice?

 Tom, et al,

Thanks for raising this issue.  A few thoughts from one of several state ABE
directors who tried to influence the direction of that $50 million back in
1993 (or thereabouts).

By (about) 1993, the R&D initiatives at USDOE (DAEL/PAS and OERI) and NIFL
were starting to gell.  We were concerned that what appeared to be emerging
were a series of small R&D efforts that were neither strategic nor tied
together in a coherent thematic way.  In our discussions with the principal
players, we asked that the three major players (DAEL National Programs,
NIFL, and NCAL -- the predecessor to NCSALL) work together on a strategic
and staged approach to R&D.  In particular, many of us argued them to tackle
assessment.  We communicated the high level of dissatisfaction practitioners
had with existing commercially available assessments, that many of us shared
their dissatisfaction, and that the challenge was beyond the capability
(read, "financial resources") of many states to tackle alone.  

Our entreaties in this regard were largely ignored.  It is true that the
state directors and others have been regularly consulted on which National
Programs and NIFL projects deserved support, but the choices have been among
several disconnected smaller initiatives that many on this list are aware
of.  There has not been a large scale coordinated effort to address the
assessment issue which is now at the heart of what is attempting to serve as
an ABE accountabilty system.  Will anyone argue that by the 1999/2000 and
2000/2001 program years that the states were prepared to provide valid,
reliable and comparable data about the educational progress students are
making?  And will anyone argue that we will do a significantly better job
when we report our program year 2001/2002 results?

A case in point that illustrates why the failure to tackle this difficult
systemic issue is so problematic.  There are two multi-year research studies
being conducted by USDOE with National Programs funding:  "What Works in
ABE" and "What Works in ES(O)L."  Each was originally funded at $1 million
per year for five years -- a total of $10 million over 5 years.  With these
studies it's important to keep in mind that without objective measures of
whether students are learning and succeeding, the answer to "what works" is
nothing more than who markets programs and services better.  In fact, the
ABE study tried to use uniformed interns to make judgements about which
programs were most likely to harbor exemplary practices after 1 or 2 visits
to a random set of programs.  [NOTE:  I and a couple of other people raised
such hell about the impending waste of scarce resources that the study was
taken off line for several months and a new contractor hired.] In the
absence of solid tools that practitioners could agree upon, both studies
spent most of the first 18 months struggling over which assessments to
"settle on," or in my words, "grit our teeth and make the best of a bad
situation."  

Equipped For the Future is a bright spot in this otherwise dismal landscape.
By starting with "what do adults want/need to know and be able to do," EFF
has tackled the real first step in developing and achieving consensus around
adult ed assessments.  For assessments to be valid and reliable, they must
align with what's being taught, i.e., curriculum.  Absent agreement about
the "universe of content," we can neither select nor construct meaningful
assessments.  Thank you for a good first step.  

But the challenge remains.  EFF is working on the assessment challenge but
they're not there yet.  Further, EFF appear's to be restricting its work to
performance assessment only.  While many of us agree that performance
asssessment can be the most promising and meaningful approach to student
assessment in adult ed, it is highly unlikely that all the challenges in
achieving validity, reliability and comparability can be resolved in a way
that is affordable in both time and money for our programs.  Some of us are
concluding that at least for the present, we will need a mix of performance
and paper and pencil assessments.  No such options appear to be emerging
nationally.

So what does this do to a state like Massachusetts?  We have convened a
"performance accountability working group" of very diverse and committed
practitioners and experts to help resolve the challenges presented by
performance accountability -- with a particular emphasis on assessment of
learning gains.  We have watched about $50 million expended on adult ed R&D
and find little to nothing of use to this effort. We are struggling with the
reality that our state may need to divert a million dollars or more away
from instruction to do the development that our leaders were asked to
undertake almost a decade ago.  We're not happy about this.

I believe our national leadership understands that the funds do not belong
to them but to our field.  I believe they also seek our feedback about what
to do with these funds -- and they have.  The problem is that we speak with
so many voices that what they end up with is long "wish lists" and the
result is a watered down set of activities intended to do something for as
many items on the list as possible.  We own part of the problem Tom, I and
others are highlighting.  Our goal must NOT be to simply have input. Our
goal must be to thrash through our many different interests, identify common
ground, and speak with a single voice about the order in which our top
priorities can and should be addressed.  We rarely do this.  On those rare
occassion when we have, we've prevailed.  

Do you want to see the next $50 million spent in a way that builds a strong
foundation for our field?  Then be prepared for some hard work -- not just
in getting national leaders to listen, but to arrive at that common ground
among your colleagues.  

bob bickerton, MA director of adult ed



-----Original Message-----
From: Thomas Sticht
To: nla at lists.literacytent.org
Cc: tsticht at aznet.net
Sent: 11/8/01 11:53 AM
Subject: [NLA] Can research improve policy or practice?

Research note							November
8, 2001

Tom Sticht

Can adult literacy research improve policy or practice?

I recently attended a couple of R & D planning meetings which got me
wondering if the 50 million or so dollars that the federally-funded
organizations that have responsibility for research (National Center for
Adult Literacy, National Center for Adult Learning and Literacy,
National Institute for Literacy, Division for Adult Education and
Literacy) have spent in the last 10-12 years have made much improvement
in either policy or practice. 

While all these organizations have been pursuing research and publishing
reports,  non-addressed issues have arisen that deal with major
consequences for the adult literacy education field. 

Scale of Need. Important  questions of the scale of need for adult
literacy education have arisen, with the National Center for Education
Statistics producing a report by the former director of the National
Adult Literacy Survey (NALS) saying that the NALS used the wrong methods
and that the scale of need is only half of what was earlier said.
Despite this, R & D organizations go on using the NALS data as though
they are valid indicators of adult's literacy skills in the USA and
elsewhere. There is also a huge gap between the numbers of adults
sampled on the NALS who thought they had a reading problem, some 95
percent thought they did not, and the reports of literacy problems
derived from the NALS standardized tests. This would seem to pose a
major problem in recruitment and increasing participation in adult
literacy education. If people don't think they have a literacy problem,
why should they seek literacy education as a solution? We need some
solid research about how many adults need basic skills instruction and
how many want it. I haven't found any research institute expressing
concern about this or doing anything to educate the field about all
this. But perhaps I am just not aware of what R & D centers are doing
about this and someone will inform me. 

Access and Participation. U. S. Department of Education officials report
that enrollments in the Adult Education and Literacy System of the
United States plummeted from over 4 million in 1997 to around 2.9
million in 2000. This is a drop of over 25 percent,  but I haven't found
any research institutes expressing concern about this, even though all
of the federally funded R & D organizations  (NCAL;NCSALL; NIFL;DAEL)
have had participation and retention as one of their major interests. 

Nature of Provision.  To my knowledge no R & D center has stated concern
about the proliferation of non-validated ideas about literacy practice
and "reform" expressed in reports from  government funded organizations,
including the R & D centers themselves. A variety of reports from these
organizations tell teachers that they and their students can benefit
from the new knowledge contained in the reports. Yet they offer very
little, if any, concrete, convincing  evidence that some
teaching/learning problems are solved or improved by anyone who
possesses the knowledge given in the numerous reports.  

Accountability.  Numerous issues regarding the National Reporting System
and its encouragement of the use of standardized tests that almost all
in the R & D field acknowledge are inadequate measures of adult learning
in adult literacy education programs have been raised in various
refereed journals, literacy newsletters,   and internet lists, including
the NLA  list.  Yet  R & D institutes have had hardly a word to say
about this, though NCSALL published a report by Juliet Merrifield that
raised a number of issues that I have not seen addressed yet by NCSALL
or any other federal adult research organization. But perhaps the
National Academy of Science committee formed recently by DAEL will
address many of these issues. 

Another aspect of accountability is the need for data regarding the
returns to investments in adult education and literacy development in
many areas. I give numerous speeches advocating for adult literacy
education in different nations each year and I repeatedly hear about the
need for ROI information.  Can adult education produce better health
care for adults and their children and produce savings in medical costs?
Does it produce savings in early childhood, compensatory education
costs, does it produce savings in training costs in business and
industry? Does it produce returns in increased productivity at work? It
is very difficult to find "hard" evidence to argue for support for the
AELS, yet there does not seem to be much such evidence being sought in
the R & D centers for adult education and literacy research that would
permit strong ROI arguments for advocating for increasing funds for
adult education and literacy development. 

Finally, in the meetings to plan future research agendas for adult
literacy and family literacy education that I recently attended I did
not hear much to lead me to suppose that the next decade will bring much
more by way of R & D that can actually improve policy or practice. For
one thing, there was no actual policy or practice problems or issues
that were identified as existing in some real place or in some real
policy that needed research to inform its change. Instead of considering
real problems of policy or practice in some real contexts, the R & D was
placed in a decontextualized frame and discussed not as problems but
rather as academic topics that need to be researched to "fill in the
gaps" in "our" knowledge, as though there are such metaphorical "gaps"
which "we" can just "plug" with a little R & D.  

The advantage of the topic approach to R & D is that it permits endless
theses and dissertations by graduate students in academic institutions
because as is well known, there is no end to the exploration of a topic.
Unfortunately, in my experience, there is also a very small track record
for such topic-oriented  R & D to solve genuine practical problems in
adult literacy education in some real place and real time (or any other
problems of education, witness the billions of dollars spent on K-12 R &
D over the last quarter century and the present state of education). 

Over the years I have been a strong advocate for R & D in adult literacy
education. But right now, the lack of any major responses to the
national problems I've mentioned above, and the lack of any evidence of
effectiveness in solving important adult literacy education problems in
various R & D reports I have recently seen from federally sponsored and
supported research and dissemination organizations make me very
concerned that if things continue in the future as they have in the
past, the field will not find much benefit from another decade of R & D
in adult literacy education.  

But perhaps I am overly pessimistic. Perhaps readers will respond with
numerous insights into how R & D has solved very practical policy and
practice problems for them and they have some data, some convincing
evidence,  to substantiate such claims. I hope so.
_______________________________________________
NLA mailing list: NLA at lists.literacytent.org
http://lists.literacytent.org/mailman/listinfo/nla
LiteracyTent: web hosting, news, community and goodies for literacy
http://literacytent.org
_______________________________________________
NLA mailing list: NLA at lists.literacytent.org
http://lists.literacytent.org/mailman/listinfo/nla
LiteracyTent: web hosting, news, community and goodies for literacy
http://literacytent.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.literacytent.org/pipermail/nla/attachments/20011109/fe2cef90/attachment.htm