[AAACE-NLA] Re: NAAL Method Issues

[tpeace@dolphin.upenn.edu]

The point of a test of any kind is based upon the evidence presented for its
valid use in making decisions. Questions about methodology often come down to
questions about validity, so it matters greatly how the NALS defines and
supports evidence for valid use. In this sense I think we should think not so
much about the scores in and of themselves but about the validity of the use of
those scores. For example, is there a subset of scores on the Document Literacy
subtest that could be used to help predict success in using transit maps? Would
this be a valid use of this subset of scores? I have the sense that the NALS,
when used in this manner, might be more useful as a tool than if it is used as
an overall judgment of a person's "literacy skills." Literacy, even when we
narrow it to be defined as the "varied uses of printed materials", is too
conceptually and contextually rich to be encapsulated in any single test or even
battery of tests.

I think literacy tests like the NALS attempt to provide predictive value---in
other words, they help us try to figure out how well a person might use a
cluster of skills that are dependent upon manipulating various kinds of printed
materials. This is a very narrow scope for literacy---and we understand that
tests cannot (and should not) be used to measure all things at the same time.
The challenge for the various adult literacy populations and those who teach and
provide services to them lies in how well literacy skills are used and whether
this usage juxtaposes well with the demands of different situations and
contexts. Adults can be BOTH flexible AND limited in their ability to shift
strategies or approaches to novel situations with alacrity---this depends on
many internal and external factors that are hard to capture even in a
qualitative manner. Literacy, when understood as a process of using information
to "shift strategy or approach," is practically impossible to "test" for in a
systematic and uniform way, but in a sense it is tested everyday as any
individual maneuvers through the world---at work, in applying for a job, in
using transportation, in caring for the family and in engaging in personal and
communal activities.

I do not doubt that the majority of the participants I work with on a daily
basis would score at the Basic level on the NALS, if not much higher. But I note
that it is in the application of literacy skills where I begin to think
otherwise. I have many people who I am absolutely certain know what a transit
map is conceptually and know exactly how it is supposed to be used. But they
have serious problems when it comes to engaging in the task of using a transit
map to make, say, more than one connection between various pieces of
transportation. This is an ongoing issue, and these difficulties don't
necessarily diminish with time, practice or through the use of different
teaching methodologies or strategies on my part. The skill(s) may not have been
learned sufficiently and completely enough so that I can rest assured they will
be able to use ANY map on their own in the future---and this is the literacy
issue. (If they find another strategy to get where they need to be in that
particular instance that's fine, but it does not diminish the concern I still
have that if they encounter a situation where they will need to use a map, they
will "get stuck.")My point is that I think we do not relearn everything each
time we encounter something novel---we invent, reuse and combine strategies for
recognizing what is novel and what is familiar and then proceed to problem solve
from there. The swifter our ability in this "recognition and recombination", the
easier it is for us to get around in the world. 

So...the question I have is, "Would a score on the NALS' Document subtest give
me a 'rough' gauge of the ability to use certain discrete 'document literacy
skills'?" Or is this not a good use of the score, given that I should still use
other criteria along with it to help me make decisions? The point I'm making is
that it's all in how I'm going to use the score and not what the score 'states'
numerically.

Tamara Peace
Graduate School of Education, University of Pennsylvania &
Transitional Work Corporation, Philadelphia, PA
tpeace at dolphin.upenn.edu
tpeace at transitionalwork.org

Quoting Anne murr <amurr at mac.com>:

> I skimmed the Key Concepts document but could not find the percentage of
> correctly-completed tasks necessary to place a respondent into one of the 4
> NAAL categories.  Didn't the 1992 NALS study require 75 or 80% correct in
> order to be in a particular Level?  That means that persons have literacy
> skills which make them capable of completing a majority, but not a high
> enough percentage, of required tasks in that Level.  Perhaps that's why only
> 41% of persons with graduate degrees could perform in the Proficient category
> in Prose Literacy?  In other words, we read well but not well enough. 
> 
> Anne Murr
> Drake University Adult Literacy Center
> 1213 25th St.
> Des Moines, IA 50311
> 
> On Sunday, December 18, 2005, at 09:00AM,
> <aaace-nla-request at lists.literacytent.org> wrote:
> 
> >
> >Today's Topics:
> >
> >   1.  MAAL Method Issues (tsticht at znet.com)
> >   2. Re:  NAAL -- What % of the adult literacy population isthe
> >      adult lite (Merle Ayres)
> >
> >
> >----------------------------------------------------------------------
> >
> >Message: 1
> >Date: Sat, 17 Dec 2005 14:57:50 -0800
> >From: tsticht at znet.com
> >Subject: [AAACE-NLA] MAAL Method Issues
> >To: aaace-nla at lists.literacytent.org
> >Message-ID: <1134860270.43a497ee29681 at webmail.znet.net>
> >Content-Type: text/plain; charset=ISO-8859-1
> >
> >December 17, 2005
> >
> >Some Methodological Issues for the NAAL
> >
> >Tom Sticht
> >International Consultant in Adult Education
> >
> >The 2003 National Assessment of Adult Literacy (NAAL), with first results
> >released on December 15th, 2005,  is a follow-up to the 1992 National Adult
> >Literacy Survey (NALS). Like the NALS, the NAAL is based upon many
> decisions
> >about how to represent the literacy abilities of adults. In reading the
> >report of the Key Concepts involved in the methodology used in the NAAL
> >several decisions that the test developers took raised questions to me
> >about the validity of the methods for representing the literacy abilities
> >of adults. Following are some quotes from the Key Concepts report and my
> >comments about the concerns the report raised for me.
> >
> >Quote: "Like other adults, NAAL participants bring to literacy tasks a full
> >range of backgrounds, experiences, and skill levels. Like real-life tasks,
> >NAAL tasks vary with respect to the difficulty of the materials used as
> >well as the complexity of the actions to be performed. However, in order to
> >be fair to all participants, none of the tasks require specialized
> >background knowledge, and all of them were reviewed for bias against
> >particular groups." [p.3, Key Concepts]
> >
> >Comment: In this quote I see two problems with the NAAL methodology for
> >assessing adult?s literacy abilities. First, the extent to which the
> >so-called "real world" tasks are actually representative of tasks each of
> >the respondents are familiar with is not documented. This raises a question
> >of the "ecological validity" of the tasks, especially when the materials
> >being used and the tasks being called for are used with adults with
> >geological and cultural differences as far apart as Native Americans on
> >remote, rural enclaves as well as with residents of Manhattan in New York
> >Cty.
> >
> >The methodology as described also violates one of the most important
> >findings about reading that much research has established: ones specialized
> >background knowledge about things read has a large effect on the person?s
> >comprehension and ability to perform tasks with the material read. A
> >hallmark of adulthood is that following formal schooling, adults go on to
> >specialize in areas related to their work and out-of-school activities,
> >such as participating in the governance of a social organization, engaging
> >extensively in hobbies and so forth. Hence to insist on tasks for which
> >adults lack "specialized background knowledge"  violates a major aspect of
> >the development of literacy in adulthood, that is, the development of
> >specialized knowledge,  and can lead to greatly underestimating adults?
> >abilities as readers in their particular life contexts. This would tend to
> >be more and more important the longer adults have been out of formal
> >schooling.
> >
> >Finally, the methodology of applying tasks in which the complexity of
> >actions called for, coupled with the failure to honor adults? contextual
> >knowledge, seems likely to account to a significant extent for the sudden
> >drop in performance of adults over the age of 50 years. Research has shown
> >that the NALS/NAALS-type tasks are positively correlated with working
> >memory, and decades of psychological and geriatrics research has
> >demonstrated that the older adults get the less efficient their working
> >memories become. Hence it is questionable whether the NAAL possesses
> >ecological validity across the age range, as well as across "real world"
> >tasks, geological, and sub-cultural contexts.
> >?..
> >Quote: "Each performance level represents a continuum of abilities
> >Although certain tasks can be characterized as typical of each performance
> >level (as shown in
> >table 2, on the previous page), it is important to remember that the tasks
> >at each level extend across
> >a certain range of difficulty and therefore require a continuum of
> >abilities. For example, the new Basic level of document literacy
> >encompasses scores ranging from 205 to 249. Adults with a score of 205 (the
> >lowest score included in the level) have a 67 percent rate of success with
> >the easiest task at that level, while adults with a score of 249 (the
> >highest score included in the level) have a 67 percent rate of success with
> >the most difficult task at that level. This means that adults at the high
> >end of the Basic level have an even higher rate of success with some of the
> >level?s easiest tasks. Moreover, these adults have a fairly high rate of
> >success with some of the tasks at the low end of the Intermediate level,
> >even though the rate is below 67 percent. "[p.17 Key Concepts]
> >?..
> >Quote: Regardless of the specific criteria used to establish performance
> >levels, adults at every level have some probability of performing any task
> >correctly. Therefore, it is not correct to say that adults at a certain
> >performance level are "not able to do" tasks at higher levels. These adults
> >are, however, less likely to succeed with such tasks." [p.18 Key Concepts]
> >
> >Comment: The two preceding quotes indicate that even though adults may be
> >classified as being in one level, such as Below Basic, they may actually be
> >able to do tasks at higher levels with some greater than zero probability. 
> >This raises the question as to just how we are to think about adult 
> >literacy competence. In a person?s mind, one probably does not have
> >literacy ability segregated by "tasks" into "levels" and hence if they
> >could do some fairly complex task, even though they could not do 67 out of
> >a hundred of such tasks, they might think themselves fairly competent at
> >reading. This, plus the considerations given above regarding "ecological
> >validity" may account for why some 95 percent of adults in the earlier NALS
> >thought they read well or very well. In turn, this may account why so few
> >show up at basic skills programs for help with their reading. Maybe they
> >actually are more competent than these assessments indicate.
> >
> >A final comment:  The NAAL provides three different scales: Prose,
> Document,
> >and Quantitative literacy and each scale has different score ranges defined
> >for the four different ability levels: Below Basic, Basic, Intermediate,
> >and Proficient. This arrangement tells us little about how we should think
> >about an individual?s overall literacy ability summed across all three of
> >these scales. Conceivably someone could be just below the Intermediate
> >level on the Prose scale but above the Intermediate level on the Document
> >scale, and perhaps even above the Intermediate level on the Quantitative
> >scale. How should this person?s overall literacy ability be thought about?
> >
> >Thomas G. Sticht
> >International Consultant in Adult Education
> >2062 Valley View Blvd.
> >El Cajon, CA 92019-2059
> >Tel/fax: (619) 444-9133
> >Email: tsticht at aznet.net
> >
> >
> >
> >
> >------------------------------
> >
> >Message: 2
> >Date: Sun, 18 Dec 2005 03:24:48 +0000
> >From: "Merle Ayres" <merleayres at hotmail.com>
> >Subject: Re: [AAACE-NLA] NAAL -- What % of the adult literacy
> >	population isthe	adult lite
> >To: aaace-nla at lists.literacytent.org
> >Message-ID: <BAY102-F347656C19B9E804010D14CC03C0 at phx.gbl>
> >Content-Type: text/plain; format=flowed
> >
> >I wouldn't like to throw a wrench in all this statistics but, scores and 
> >goals are meaningless in adult literacy if the outcome is only the score 
> >results. Can a basic GED. or diploma help the generation xers work a 8 or 9
> 
> >hour day? Our work ethic is missing in the younger generation. College does
> 
> >not give an automatic work ethic,. I always encouraged this in school that 
> >working on something hard was good. I don,t see the stick it to iveness 
> >today in the new work force.
> >
> >Merle Ayres
> >412 8th st. North
> >Humboldt,Iowa 50548
> >Tel.1-515-332-4630
> >Fax 515-332-1738
> >
> >
> >
> >
> >>From: John Comings <comingjo at gse.harvard.edu>
> >>Reply-To: National Literacy Advocacy List sponsored by 
> >>AAACE<aaace-nla at lists.literacytent.org>
> >>To: National Literacy Advocacy List sponsored by 
> >>AAACE<aaace-nla at lists.literacytent.org>
> >>Subject: Re: [AAACE-NLA] NAAL -- What % of the adult literacy	population 
> >>isthe adult literacy population
> >>Date: Sat, 17 Dec 2005 09:59:11 -0500
> >>
> >>I haven't checked but I think the "high school" in this case is finished 
> >>high school but did not go on to postsecondary education. Transition into 
> >>some form of post secondary education or its equivalent in skilled job 
> >>training is essential to compete in today's job market.
> >>
> >>--On Thursday, December 15, 2005 3:11 PM -0500 Hal Beder 
> >><hbeder at rci.rutgers.edu> wrote:
> >>
> >>>If you look at the average scores [p14] for high school and GED they are
> >>>virtually the same, 262-260 for prose.  These scores fall at the high end
> >>>of the basic scale. Thus if we want to get them up to secondary education
> >>>proficiency we have to shoot for both below basic and basic.  So the new
> >>>NAAL makes the case even better than NALS which labeled the levels with
> >>>numbers.
> >>>
> >>>
> >>>At 12:12 PM 12/15/2005, you wrote:
> >>>>Hello Hal,
> >>>>
> >>>>Here's one rationale:
> >>>>
> >>>>Since most people would agree now that one needs at least high school
> >>>>(or GED) level skills to qualify for post secondary education, and
> >>>>that some amount of post secondary education is needed to attain
> >>>>family self sufficiency, if the purpose of adult literacy education
> >>>>is to enable family self-sufficiency, then our mission is to serve 42
> >>>>% of the American adult population.  There will, of course, be
> >>>>confusion around calling this "the adult literacy population" for
> >>>>people who define literacy narrowly as basic competence in reading
> >>>>and writing, and who do not take into account what reading, writing,
> >>>>numeracy and other skills are needed for family self-sufficiency.
> >>>>
> >>>>David J. Rosen
> >>>>Adult Literacy Advocate
> >>>>DJRosen at theworld.com
> >>>>
> >>>>
> >>>>On Dec 15, 2005, at 10:40 AM, Hal Beder wrote:
> >>>>
> >>>>>The 1993 NALS broke literacy scores into five levels and it became
> >>>>>commonly accepted that the adult literacy population was
> >>>>>represented by levels one and two, about 46-48% of the adult
> >>>>>population.  Although this figure may have been counter-intuitively
> >>>>>high, it was used to justify increased funding in many states and
> >>>>>over the years it became the "official" adult literacy figure in NJ
> >>>>>and elsewhere.  The newly released NAAL breaks the scores into four
> >>>>>levels rather than five: below basic, basic, intermediate and
> >>>>>proficient.  The percent of the adult population for below basic is
> >>>>>about 16% if you average prose, document and quant while the
> >>>>>percent for basic is about 26%.  So here is the huge question.
> >>>>>What percent of the adult population constitutes the adult literacy
> >>>>>population now?  If you say that the mission of adult literacy is
> >>>>>to bring adults up to the basic level, it includes only the below
> >>>>>basics,  16%, as compared to 46-48% from the 1993 NALS.  If you say
> >>>>>it includes the basics as well as the below basics, it's about 42%,
> >>>>>but what is the rationale for doing that?  I can't think of
> >>>>>anything convincing.  How do we interpret the new NAAL to those
> >>>>>whose support for adult literacy we depend on?
> >>>>>
> >>>>>Rutgers University
> >>>>>Graduate School of Education
> >>>>>10 seminary Pl.
> >>>>>New Brunswick, NJ 08901
> >>>>>732-932-7496 ext. 8213
> >>>>>_______________________________________________
> >>>>>AAACE-NLA mailing list: AAACE-NLA at lists.literacytent.org
> >>>>>http://lists.literacytent.org/mailman/listinfo/aaace-nla
> >>>>>LiteracyTent: web hosting, news, community and goodies for literacy
> >>>>>http://literacytent.org
> >>>>
> >>>>_______________________________________________
> >>>>AAACE-NLA mailing list: AAACE-NLA at lists.literacytent.org
> >>>>http://lists.literacytent.org/mailman/listinfo/aaace-nla
> >>>>LiteracyTent: web hosting, news, community and goodies for literacy
> >>>>http://literacytent.org
> >>>
> >>>Rutgers University
> >>>Graduate School of Education
> >>>10 seminary Pl.
> >>>New Brunswick, NJ 08901
> >>>732-932-7496 ext. 8213
> >>>_______________________________________________
> >>>AAACE-NLA mailing list: AAACE-NLA at lists.literacytent.org
> >>>http://lists.literacytent.org/mailman/listinfo/aaace-nla
> >>>LiteracyTent: web hosting, news, community and goodies for literacy
> >>>http://literacytent.org
> >>
> >>
> >>
> >>John Comings, Director
> >>National Center for the Study of Adult Learning and Literacy
> >>Harvard Graduate School of Education
> >>7 Appian Way
> >>Cambridge MA 02138
> >>(617) 496-0516, voice
> >>(617) 495-4811, fax
> >>(617) 335-9839, mobile
> >>john_comings at harvard.edu
> >>http://ncsall.gse.harvard.edu
> >>
> >>_______________________________________________
> >>AAACE-NLA mailing list: AAACE-NLA at lists.literacytent.org
> >>http://lists.literacytent.org/mailman/listinfo/aaace-nla
> >>LiteracyTent: web hosting, news, community and goodies for literacy
> >>http://literacytent.org
> >
> >
> >
> >
> >------------------------------
> >
> >_______________________________________________
> >AAACE-NLA mailing list
> >AAACE-NLA at lists.literacytent.org
> >http://lists.literacytent.org/mailman/listinfo/aaace-nla
> >
> >
> >End of AAACE-NLA Digest, Vol 31, Issue 28
> >*****************************************
> >
> >
> _______________________________________________
> AAACE-NLA mailing list: AAACE-NLA at lists.literacytent.org
> http://lists.literacytent.org/mailman/listinfo/aaace-nla
> LiteracyTent: web hosting, news, community and goodies for literacy
> http://literacytent.org
> 


--