[NLA] Government Misinformation - Summing Up

[Thomas Sticht]

Research Note						12 November 2002

Tom Sticht
International Consultant in Adult Education

NALS-Based Misinformation From the Government: Summing Up

On  November 1st I posted the first of three Research Notes pointing to
misinformation being disseminated by government agencies or government
funded agencies based on faulty data from the 1993 National Adult Literacy
Survey (NALS).

Research Note #1 (1 Nov 02).

In this note I claimed that there was  NALS-Based Misinformation at
USED/OVAE/DAEL. I noted that the USED web site for OVAE/DAEL states that:

Quote: "The National Adult Literacy Survey estimates that about 90 million
adults in the United States may lack the literacy skills needed to succeed
in the economy of the future. End Quote

I then pointed out that this was not correct on two counts. First, in the
original 1993 report on the NALS, on page xviii the NALS researchers
rhetorically asked whether the findings answered the critical question,
"Are the literacy skills of America’s adults adequate 
 to ensure
individual opportunities for all adults, to increase worker productivity
or to strengthen America’s competitiveness around the world?" End Quote.
The answer given by the NALS researchers: Quote:"Because it is impossible
to say precisely what literacy skills are essential for individuals to
succeed in this or any other society, the results of the National Adult
Literacy Survey provide no firm answers to such questions." End Quote.
Hence there is no way that the NALS could estimate that about 90 million
adults in the U. S. may lack the literacy skills needed to succeed in the
economy of the future. It could not even say whether the literacy skills
met the needs of the present (that is, 1993).

Second, and even more troubling with the OVAE/DAEL misinformation, is that
it occurs even though it has been known to ED officials for some time that
the final NALS Technical Report of January 2001 includes chapters by Dr.
Andrew Kolstad, the former director of the NALS project at the NCES, which
show that the NALS used arbitrary statistical methods which greatly
overstated the numbers of adults with poorly developed literacy skills. In
fact, using the new statistical methods he recommends, because they
produce the most valid estimates of skills, the percentages of adults in
the lowest two levels of literacy that the NALS uses fall by more than
half! Thus the OVAE/DAEL statement of adults with "at risk" literacy
skills would drop from 90 to less than 45 million adults.

The Response From USED/OVAE/DAEL: None

Research Note #2 (4 Nov 02).

Here I called attention to NALS-Based Misinformation at the NIFL:
Partnership For Reading pages at the NIFL web home page. I noted that the
report on reading comprehension contained information that is not correct.
Under a section at
(www.nifl.gov/partnershipforreading/adult_reading/comprehension/compassess5.html)
called Quick Summary of the Research, the report states:

Quote: "A recent, large-scale study of adults' reading comprehension (the
National Adult Literacy Assessment (sic), or NALS) provides information
about adults' reading comprehension that is more reliable than the
information we have about adults' fluency and vocabulary. Results from the
NALS indicate that most ABE learners will have difficulty integrating and
synthesizing information from any but the simplest texts.." End Quote

I then noted that the problem with the foregoing is that, using the
Partnership’s definition of reading comprehension, Ouote"Reading
comprehension can be described as understanding a text that is read, or
the process of constructing meaning from a text (National Reading Panel,
p. 4-5). " End Quote, the NALS did not assess reading comprehension, even
in the so-called Prose comprehension scale. Instead, it assessed the
performance of tasks ranging in cognitive complexity from low to high, all
of which required the use of two or more "skills," thereby rendering it
impossible to say for certain just what was being measured, a point made a
decade ago (May 1992 Matching literacy testing with social policy.
National Center on Adult Literacy) about such "real world," functional
literacy tasks by Richard Venezky, a recent recipient of a NIFL grant to
study reading. Nor is it possible to say why it was that adults could not
perform the tasks, e. g, whether they could not construct meaning
(comprehend) from the language used in the tasks, or if their information
processing abilities in other skills posed problems for them.

I also called attention to other research by the former director of the
NALS project at the National Center for Education Statistics (NCES) and
others that questioned the construct and standards (i.e. response
probability)  validity of the NALS.

The Response From NIFL: None

Research Note #3 (7 Nov 02).

In this note I called attention to misinformation on the home web page of
the federally-funded National Center for the Study of Adult Learning and
Literacy (NCSALL) which stated: Quote" More than 40 percent of working-age
adults in the United States lack the skills and education needed to
succeed in family, work, and community life today. By strengthening
practice and policy, NCSALL helps these adults gain an opportunity to
achieve their full potential." End Quote

I asked rhetorically, Why is this statement misinformation? Then I
answered, Because there is no known methodology for determining what the
skills and education needed to succeed in family, work, and community life
across the nation are [recall from Research Note #1 that the NALS
researchers had made this point themselves] . It is even more difficult to
identify what it means to "succeed" in family, work, and community life.

I concluded that, clearly, the claim that Quote "More than 40 percent of
working-age adults in the United States lack the skills and education
needed to succeed in family, work, and community life today"  cannot be
substantiated and represents yet another example of federally-funded
misinformation about the skills of America’s adults.

The Response From NCSALL: Dr. John Comings, Director of NCSALL,  confirmed
my claim of the dissemination of  misinformation by acknowledging that the
NCSALL research team knew all along about the validity problems of the
NALS but they ignored these problems and used the NALS data anyway in
forming their estimate of 40 percent of the working age adults who "lack
the skills and education needed to succeed in family, work, and community
life today." But since this claim is based on invalid data from the NALS,
it cannot be considered accurate. It is misinformation.

Comment

Taken together, these three Research Notes point to a continued use by the
government and government-funded agencies of the results of a major survey
which is now acknowledged by experts in psychometrics, literacy, and
measurement to lack construct, standards (i.e., response probability), and
use or consequential validity. Why this continued use is so is not known
to me, but I suspect that it reflects a reluctance to admit the
expenditure of millions of dollars on such a colossal blunder by a number
of "experts," i.e., advisory panel members, researchers, reading
specialists, psychometrists, and government officials.

My guess is that this reluctance to acknowledge the problems of the costly
NALS is compounded by the fact that the government officials of the U.S.
and contractors to the government convinced other nations to use the NALS
methods and to conduct the International Adult Literacy Survey (IALS). 
Consequently, now these other nations must defend their expenditures of
funds on what is more and more recognized as an invalid approach to
assessing adult literacy.

So through the power that these government officials possess by virtue of
their oversight of millions of taxpayers dollars, and their ability to
send those dollars to those who support them, they are able to keep the
misinformation flowing. I think that contributing to this continued flow
of misinformation is the seeming lack of any indication from the adult
literacy education field that anyone cares whether the NALS or IALS are
valid indicators of the literacy abilities of adults. What does it matter,
anyhow?





_______________________________________________
NLA mailing list: NLA at lists.literacytent.org
http://lists.literacytent.org/mailman/listinfo/nla
LiteracyTent: web hosting, news, community and goodies for literacy
http://literacytent.org