[NLA] Consequential validity

[Regie Stites]

The late Samuel Messick developed the concept of consequential validity as one of
six aspects of a unified notion validity (for all educational and psychological
measurement).  He defined the consequential aspect of validity as an appraisal of
"the value implications of score interpretation as a basis for action as well as
the actual and potential consequences of test use, especially in regard to sources
of invalidity related to issues of bias, fairness, and distributive justice" (see
Messick's essay on "Validity" in the 1989 (3rd) edition of Educational Measurement,
R. L. Linn (Editor), New York: Macmillan).

Including consequences of test interpretation and use as explicit considerations in
the validation of educational tests and assessment is still a relatively new idea
in psychometrics, and something that needs more attention from test developers and
from test users.

The latest edition (1999) of the "Standards for Educational and Psychological
Testing" (published jointly by the American Educational Research Association,
American Pscyhological Association, and the National Council on Measurement in
Education) incorporates consequential validity in test validation.  But, in the
section introducing standards for validity, they make a distinction between
"intended and unintended consequences of test use ... that can inform validity
decisions"  on the one hand, and "evidence that may inform decisions about social
policy but falls outside the realm of validity" on the other (p. 16).   In other
words, not every problem of differential consequences from test use (in making
decisions about employment or about academic promotion or selection, for example)
is a problem with the validity of the test.  Some problems with the consequences of
test use are strictly matters of social policy -- bad decisions made by people
interpreting test results (or research results), not flaws in the design of the
tests (or research methods).

Regie Stites, educational researcher

Elsa Auerbach wrote:

> Can anyone explain the concept of consequential validity?   I think it's
> relevant to Freire and other social change pedagogies and research.  Elsa
> Auerbach
>
> PDRNRI at aol.com wrote:
>
> > A few short observations from a quick read of Catherine's meaningful letter:
> >
> > 1.  I would like to hear from other researchers on this list on how the
> > research field is grappling with the govt.'s emphatic prioritizing of a
> > rather one-dimensional approach to research.  It seems to me that there are a
> > great many researchers in the field today who seek to approach learning and
> > development as something "dialogic, self-reflective, political, ethical and
> > creative" (Catherine's terms).  How are you responding to this?
> >
> >  2.  Reflecting on Catherine's important, ongoing reminder that we view our
> > society as a "maturing democracy" - it occurs to me that for most of our
> > history (certainly the 20th century) the predominant question challenging the
> > democratic process in the US. has been "who gets to participate?" .  The
> > dilemma of modern adult education, it seems to me, is partly connected to a
> > shift (or the lack thereof) in the popular discourse from  "who gets to
> > participate?" to "how do we prepare ourselves to participate?"  I suspect
> > that while many of the folks on this list -- who have thought hard about
> > democratic participation for so long -- see this question clearly and have
> > developed very carefully considered answers.  But I'm less sure that the
> > popular thinking on this question has advanced a great deal -- partly because
> > issues around "who gets to participate" are not fully resolved, partly
> > because actual preparation for participation is more threatening to those who
> > exert the most power, and perhaps partly because many of those who have had a
> > long-standing right to participate have been effectively dissuaded from doing
> > so, having bought into the idea that particaption ends, rather than begins,
> > at the ballot box.  In that case, the issue of preparing for more effective
> > participation would simply not be on the radar among those who can apply
> > political pressure through popular opinion.
> >
> > 3.  More good books, for the same purposes, I think, as Catherine's:
> > Making Spaces (Sissel and Sheared, eds.)  In Defense of the Lifeworld
> > (Welton, ed.) -- and for teacher development purposes, Becoming a Critically
> > Reflective Teacher (Brookfield)
> >
> > David Hayes
> > _______________________________________________
> > NLA mailing list: NLA at lists.literacytent.org
> > http://lists.literacytent.org/mailman/listinfo/nla
> > LiteracyTent: web hosting, news, community and goodies for literacy
> > http://literacytent.org
>
> _______________________________________________
> NLA mailing list: NLA at lists.literacytent.org
> http://lists.literacytent.org/mailman/listinfo/nla
> LiteracyTent: web hosting, news, community and goodies for literacy
> http://literacytent.org
-------------- next part --------------
A non-text attachment was scrubbed...
Name: regie.stites.vcf
Type: text/x-vcard
Size: 389 bytes
Desc: Card for Regie Stites
Url : http://lists.literacytent.org/pipermail/nla-nifl-archive/attachments/20020405/26cd21b8/regie.stites.vcf