[Fwd: Re: [NLA] Evidence-based practice in Adult Literacy Education](LONG!)

[Ralf St.Clair]

This may be interesting to people. It is a note I wrote (to myself) 
after attending the NIH/NIFL presentation in Houston. I don't treally 
know what to do with it, so I'll share it with you folks in the hope 
that it may be valuable. Sorry it's so long!

Ralf



Research for practice: What is meant by scientifically based research 
in education?
Ralf St.Clair, Texas A&M University, April 2002

There has been a lot of discussion recently about finding more 
effective ways to teach literacy to adults and children. In family 
literacy, for example, there is now a requirement to use a "research 
based" early childhood reading curriculum. At the same time agencies 
in Washington have become interested in funding "scientifically 
based" research as a decision making tool (a good example is the 3.8 
million dollars offered by the National Institute of Health and the 
National Institute for Literacy in 2002-2003). These developments 
have already made a difference to many of our practices and are 
likely to become more influential over time. So what do the terms 
"research based" and "scientifically based" refer to, and how do we 
know it when we see it?

In order to explore these questions, this brief paper summarises the 
report of a Department of Education working group conference held on 
February 6th, 2002. The title of the conference was "The use of 
scientifically based research in education," and participants 
included Susan B. Neuman (Assistant Secretary for Elementary and 
Secondary Education) and nine presenters. The two aims of the 
conference were "to explore the logic of scientifically based 
evidence" and to discuss how "we begin to put this into practice."

Valerie Reyna's presentation raised the question of what we rely on 
if not scientific research-unfounded beliefs, anecdotes, memory? She 
argued that clinical trials (as in medicine, where people randomly 
receive either a treatment or a placebo and statistics are then used 
to test whether the treatment made a difference) are the only way to 
make a causal inference (a causes b). Below clinical trials are 
quasi-experimental models (where there is no random assignment) and 
below them comes evidence based theory (theory with some confirmation 
of the predictions they produce). Reyna then talked about the way the 
findings of research would be put into practice. Overall the thrust 
of her presentation was to emphasise the value of conventional 
scientific method, and her hope for a time when "the use of 
scientific research as a basis for educational practice will become 
routine."

Michael Feuer of the National Academy of Science argued that "science 
is intendedly rational, it is disciplined, it is honest, it is open, 
we aspire to a kind of dispassionate, politically neutral 
distillation of evidence to make decisions. However, he also pointed 
out that "what scientists themselves often acknowledge is that there 
is a dimension of human judgement that can be missed with an 
overzealous focus on the rigours of scientific method." What 
constitutes rational decision making?

Lisa Towne laid out five principles of science lying at the heart of 
scientifically based research. They were that questions should be new 
and different while also being significant and empirical-meaning they 
can be answered by observation. Data must be linked to theory, 
methodology must be appropriate, and the overall research project 
must have a coherent chain of reasoning. Research should be 
replicable (able to be repeated with the same results) and 
generalisable (meaning the results can be applied in many settings). 
The final principle is the transparency of the scientific enterprise. 
Towne also highlighted several challenges to the application of this 
type of research to education. She emphasised the central place of 
politics and values in education-in American society schools are 
strongly grounded in notions of democracy. There are also ethical and 
relationship issues in educational research.
Stephen Raudenbush followed up by arguing for the importance of 
practitioner judgement. "There are many terrifically important 
questions for educational policy that are not causal," suggesting 
that the strong model of scientific research may not always be 
appropriate. Qualitative research (unscientific in terms of the 
principles laid out above) can generate good ideas to take into the 
field and test-"those ideas coem from up close, careful study of 
expert practitioners in real settings . . ." Raudenbush closed by 
expressing his hope for a balanced approach to educational research 
and the development of really high quality work.

Reflections
The most important implication to draw from these discussions is the 
strength of the boundary between positivist research on one side and 
interpretivist research on the other. Stating the difference as 
strongly as possible, the positivist model emphasises quantitative 
methods as the path to "truth" and the interpretivist model believes 
each fact has many meanings, which can be partially revealed through 
qualitative methods. While it is unusual to come across such extreme 
positions (for example, many scientists view truth as highly 
contingent) there is nonetheless a fundamental tension between the 
two ways of looking at the world and at the use of research.

It seems from the conference discussion that there is some concern 
about relying completely on a strongly positivistic model of 
educational research. The speakers who mentioned this concern, 
however, seemed to be less certain about what to do with questions of 
values or politics. Using qualitative forms of research to identify 
the ideas to be scientifically tested appears to be a relatively 
common approach in current discussions (including the NIH/NIFL 
projects mentioned above), and some would claim that this deals with 
issues of value. This perspective holds that good ideas are chosen in 
the qualitative stage, and then the objective quantitative methods 
step in to provide "real" evidence. To take this approach is to move 
qualitative research into the role of handmaiden to the quantitative, 
scientifically based research, and this is a logical error. If 
quantitative research cannot deal effectively with issues of value, 
and values are at the heart of education, then a quantitative 
approach is fundamentally incapable of addressing the important 
questions. Developing "good ideas" through a rich mix of anecdote, 
professional experience, and participatory method and then reducing 
it to a hypothesis suitable for a clinical trial is both demeaning to 
interprevitist research and compromises the contribution of 
quantitative work.

Any well trained researcher begins with the question to be addressed. 
Only then will they decide how to address it, and the choice of 
method will vary with the question. It is extremely unusual for the 
same question to be legitimate in both broad models of research. In 
other words, generating good ideas and testign them are very 
different endeavours, and do not go together in as straightforward a 
manner as implied above.

If the definition of scientifically based research outlined at this 
conference is accepted as the norm for educational research (the 
phrase "gold standard" was used many times), only certain types of 
information will be seen as valuable. There will be gaps in the 
practices we can develop or defend, and one of the most important 
will be local knowledge. Over many years programs build knowledge of 
what works in their community-knowledge often unspoken, or tacit. In 
the face of scientifically based research such knowledge does not 
count-in that it is not generalisable  or replicable, and may well 
not be empirical in the sense of being directly observable. If only 
clinically  tested knowledge counts, then those with the resources to 
conduct such trials get to define what is true.

Conclusion
Moves towards greater quality-and funding-in adult literacy research 
are to be applauded, but the applause must remain muted. The question 
of what makes good research is far from settled however much the 
current political players would like to claim that it is. It is still 
open to discussion whether emphasising scientifically based 
approaches derived from natural and medical sciences is at all 
helpful. Education is a human science, and the human sciences embrace 
many forms of data collection and analysis, with many valuable 
insights to offer. Science is, after all, about the generation of 
knowledge, and as Raudenbush argued  "what's really, in the final 
analysis, scientific, is what the community of scientists say is 
scientific." In education what constitutes science is still wide 
open-diversity in research, in knowledge, in values, in politics, is 
not the weakness of our field, but its strength.
_______________________________________________
NLA mailing list: NLA at lists.literacytent.org
http://lists.literacytent.org/mailman/listinfo/nla
LiteracyTent: web hosting, news, community and goodies for literacy
http://literacytent.org