[NLA] Evidence-based literacy misinformation-Part 2

[Thomas Sticht]

Research Note                     4 November 2002

Tom Sticht
International Consultant in Adult Education

Literacy Misinformation From the U.S. Government Part 2: National
Institute for Literacy

With the strong desire to promote scientific, evidence-based education for
literacy development across the life span, the U. S. Government is
providing extensive information about reading and literacy in sources on
the internet. However, a recent visit to two federal government web sites
revealed that instead of providing valid information about adult literacy
and reading,  both the U. S. Department of Education, Office of Vocational
and Adult Education (OVAE), Division of Adult Education and Literacy
(DAEL) and the National Institute for Literacy (NIFL) are presenting
information based on the 1993 National Adult Literacy Survey (NALS) that
another U. S. Department of Education Agency, the National Center for
Education Statistics (NCES), which conducted the NALS, has indicated is
invalid. This message presents the misinformation from NIFL.  An earlier
NLA post of 1 Nov 02 presented the misinformation from USED/OVAE/DAEL.

NALS-Based Misinformation at the NIFL: According to the Partnership For
Reading pages at the NIFL web home page, the Partnership aspires to
present evidence-based practices for reading instruction, including adult
reading instruction.

However, while perusing the report on adult reading instruction at the
NIFL web site, I came across information about reading comprehension that
is not correct. Under a section at
(www.nifl.gov/partnershipforreading/adult_reading/comprehension/compassess5.html)
called Quick Summary of the Research, the report states:

Quote: "A recent, large-scale study of adults' reading comprehension (the
National Adult Literacy Assessment (sic), or NALS) provides information
about adults' reading comprehension that is more reliable than the
information we have about adults' fluency and vocabulary. Results from the
NALS indicate that most ABE learners will have difficulty integrating and
synthesizing information from any but the simplest texts.." End Quote

The problem with the foregoing is that, using the Partnership’s definition
of reading comprehension, Ouote"Reading comprehension can be described as
understanding a text that is read, or the process of constructing meaning
from a text (National Reading Panel, p. 4-5). " End Quote, the NALS did
not assess reading comprehension, even in the so-called Prose
comprehension scale. Instead, it assessed the performance of tasks ranging
in cognitive complexity from low to high, all of which required the use of
two or more "skills," thereby rendering it impossible to say for certain
just what was being measured, a point made a decade ago (May 1992 Matching
literacy testing with social policy. National Center on Adult Literacy)
about such "real world,"functional literacy tasks by Richard Venezky, a
recent recipient of a NIFL grant to study reading. Nor is it possible to
say why it was that adults could not perform the tasks, e. g, whether they
could not construct meaning (comprehend) from the language used in the
tasks, or if their information processing abilities in other skills posed
problems for them. In short, while reading comprehension was clearly
necessary to perform the NALS Prose tasks, it is equally clear that it was
not sufficient.

Indeed, the research by Irwin Kirsch, Ann Jungeblut, and Peter Mosenthal
summarized in Chapter 13 of the final NALS Technical Report of January
2001 indicates that in addition to comprehension, the Prose scale of
difficulty increased from low to high difficulty in part because of
information search and locate skills that involve the holding of more and
more features of information in working memory to be used in locating
relevant information in more and more complex texts. This type of
information processing does not fit the definition of reading
comprehension given in the Partnership for Reading as constructing meaning
from text.  Consequently it is incorrect to say that the NALS
Quote:"provides information about adults' reading comprehension" End Quote
as scientific, evidence-based information for guiding adult reading
instruction practice.

Further, the issue of just what it is that the NALS so-called "literacy"
scales actually measure was raised in Chapter 12 of the final Technical
Report by Donald Rock and Kentaro Yamamoto. Following extensive
statistical (factor) analyses, they concluded that about 60 percent of the
reliable variance of the Prose scale was NOT independent of a general
academic "G" factor. This is the type of factor that is often called
"verbal intelligence". Hence it is not clear to what extent the NALS Prose
scale measures some form of "verbal intelligence" or some form of "prose
literacy" or other, unknown, things, too (over a third of the variance was
error variance). Similar difficulties of interpretation hold for the NALS
Quantitative and Document scales, too. And, indeed, Rock and Yamamoto
report correlations among the three literacy scales ranging from +.87 to
+.91. This suggests that these scales may largely be measuring the same
thing(s), and raises the question not only of what is it that the scales
are measuring, but also the question of why, given the expense of test
construction and survey administration (in excess of $10 million dollars),
three scales are necessary.

What ever it is that the NALS scales measure, it is clear that what is
stated at the NIFL Partnership for Reading web page, i.e., that the NALS
"provides information about adults' reading comprehension
" is incorrect
and is not based on valid, scientific, evidence-based research on adults’
abilities to construct meaning from text, which is how the Partnership for
Reading defines reading comprehension.






_______________________________________________
NLA mailing list: NLA at lists.literacytent.org
http://lists.literacytent.org/mailman/listinfo/nla
LiteracyTent: web hosting, news, community and goodies for literacy
http://literacytent.org