[AAACE-NLA] The Gold Standard

[George E. Demetrion]

(Crossposting)

The Gold Standard

In certain research circles, randomized sampling, based on a "pure"
experimental model of design, is viewed as the gold standard of social
science research.  I would suggest if there is any gold standard it would
be in the combination of the quality of question(s) asked, the design(s)
that follows to best probe into the question(s), and the actual carrying
out of any particular research project.

Advertisers, for example,  might use random sampling as a first cut to
determine how many television viewers watched a particular program. 
Assuming there was a way to concretely identify a representative
sub-sample of those viewers, product and interest surveys based on one or
more of the commercials in a given program could be further given to get
a better understanding of consumer purchasing decisions of that viewer
subset.  

That would still be a long way off between evaluating the impact on the
advertising in relationship to the motivation and actual purchasing
decisions of individuals who watched the program, though useful
information en route would be gained.  Needed from there would be
in-depth focus group analysis in order to derive a qualitative assessment
of the ways in which any particular ad  influenced the viewers.  From
there, an imaginative leap into poetics would be needed in the
advertising brain storming session in order to create new imagery that
may better tap into core symbols of what would stimulate the relevant
viewers to purchase the product or service in question.

Thus even here there is no singular research methodology that reflects an
objective gold standard of legitimate research, but a variety of
methodologies that shift in importance based on the focus of the project
throughout its various stages of evolution.  Moreover, one does not
necessarily need to start from the random sample of viewers.  Rather,
given the applied nature of the advertising research project, one might
have started from an informal in-depth sample, or even from a hunch of
the creator of the ad or the product manager of the good or service in
question.  The research project, then, would need to be further developed
to gain the desired information of linking the ad to the motives and
consumer preferences of the viewers. That would call for a diversity of
methodologies and designs to be determined by the focus of the project
itself.

 The point, after all, is to identify a good research question and then
to organize the research in order to obtain the information needed to
best answer it.  Again, given the applied nature of advertising research,
a new ad might be created on what is viewed as >sufficient< research
since ALL the data can never be collected (never mind analyzed) in any
event.  Choices and priorities have to be made all along the way, and in
getting a handle on the nature of human motivation, complex qualitative
issues need to be addressed as well as quantitative ones of measurement. 
Consequently, neither quantitative nor qualitative data are privileged
per se.

If such is the case with advertisement research,then  what about research
on adult literacy?

At least in some quarters the following hierarchy in adult literacy
research quality is asserted:

* Random sampling experimental design (the gold standard).
* Quasi-experimental design ( the silver standard).
* Correlational studies with statistical control; the bronze standard
* Correlationational studies without statistical control (the copper
standard).
* Last and least, case study analysis (the clay standard).

What stands out to me is a methodological fetishism (i.e., obsession) as
if the quality of research depends primarily on adhering as closely as
possible to the defined gold standard.  Thus, the research question, per
se, is subordinated to this more fundamental methodological requirement. 
I find that curious, to say the least, except that I understand the
politics of it, and thereby its logic.

There are a lot of good research questions that could be posed.  Given
the marginality of resources supporting adult literacy, one good research
question would focus on a longitudinal study of a select group of
students who have made significant progress in their learning over, say,
a three-year period.  Progress is a complex phenomenon, which would need
some type of definition.  What would not be needed, however, is a uniform
definition, with the hope of isolating and controlling all the variables
in order to precisely isolate one independent variable from another. 
That would not be needed as long as experimental design is not viewed as
the gold standard of research. 

 Without that albatross, different descriptions of progress as actually
experienced by students would be given legitimate play in research.  This
is one place where case-study ethnographic research would come to play
and take on a major role in the research project in laying out concretely
some of the many variables at work in the characterization of a given
case study analysis of student progress.  At that point, other
methodologies and designs would come to play in order to critically
assess the significance and relevance of the ethnographic data, but the
case study, ethnographic data would provide a critical baseline that
would describe something of how learning has taken place with a given
student population.

Some of the factors in such a proposed research project that would need
to be addressed are the following:

* First and foremost (perhaps), the definition of literacy:  Whether the
focus is primarily on reading and writing, or learning more broadly
defined through the realm of print and verbal discourse.  This would
include learning both within and outside the program, and the impact of
such learning on one's personal life and community impact.

* The nature of the instructional program and amount of class time
available.

* First-hand observations and data analysis of student work over time.

 * The background, skills, and knowledge base of students, including
reasons for their participation and sources of motivation and support.

* A detailed historical analysis of student learning, knowledge
development and skill acquisition both within and outside the program
within each of the years of the study.

* A coherent composite descriptive analysis subject to further refinement
and alternative explanation.

I've made the point in a previous posting that 100 carefully identified
case studies would be substantially superior to 100,000 quantitative
studies that did not significantly take the context of concrete learning
scenarios into account.  Allowing me some rhetorical license here, at the
very least the empirical data would be quite different.

 Question:  What would we as a field learn through a carefully developed
analysis of 100 students representing different programs, entry-level
points (age, ethnic, racial, and gender backgrounds), focusing on those
students, however variously defined, who have attained superior progress
in a three year period?  I emphasize students with superior progress in a
highly marginalized field like ours in order to gain something of the
outer boundaries of what could be accomplished through adult literacy if
it were adequately resources in the many ways that the field currently is
not so supported.  Such information, I argue, would likely yield valuable
insight on best instructional and programmatic practices.  That is at
least what I am hypothesizing.

 Sub-questions:

1.  What would be the variables that would need to be taken into account
in developing a research project on this framework?

2.  What methodologies would need to be drawn upon at what stages of the
research project in order to be both rigorous and imaginative in design
and to yield the optimal results?

3. If the result were a composite profile of the field that provided the
field with a substantial road map on how best to organize programs and
structure instruction, would the research be valuable even if it weren't
shaped by an experimental design model of research?  (Obviously there
would need to a way to generalize from the specific.  The issue is how
rigorously quantitative the sample needs to be and whether a mathematical
rather than a critical descriptive metaphor is of more value in getting
at key issues in adult literacy research and practice).

4. Why is there such a fetish over methodology when there are so many
issues of substance that needs to be addressed? (That, I hypothesize,
would illuminate key issues in the politics of literacy and the diffusion
process of how those politics are being played out on the national
landscape).

5.  How can this current federal domination of methodological fetishism
be changed, and what are the costs of doing so and not doing so?

 George Demetrion





________________________________________________________________
The best thing to hit the internet in years - Juno SpeedBand!
Surf the web up to FIVE TIMES FASTER!
Only $14.95/ month - visit www.juno.com to sign up today!