[AAACE-NLA] Instructional Engineering in the AELS

[tsticht@znet.com]

February 15, 2007

Instructional Engineering Research in Adult Literacy Education

Tom Sticht
International Consultant in Adult Education

The Office of Management and Budget (OMB) has announced its most recent
assessment of the Adult Education State Grants (SG) which provide the
Federal part of the funds for the Adult Education and Literacy System
(AELS) of the United States. For the announcement see:
www.whitehouse.gov/omb/expectmore/summary/10000180.2006.html

The OMB gave the AELS/SG an "Effective" rating and went on to say: " This is
the highest rating a program can achieve. Programs rated Effective set
ambitious goals, achieve results, are well-managed and improve efficiency."

Like other Federal education initiatives, the AELS/SG is under the pressure
of providing "evidence-based" programs for students. Given the impressive
rating of the AELS/SG by the OMB it is important to understand what
evidence was consulted by the OMB in making this assessment. This can
reveal what at least one part of the Federal government considers
sufficient evidence for determining that a program is "effective."

The OMB cites three major lines of evidence for making its determination of
the effectiveness of the AELS/SG. The first states: "*Recent evaluation
findings and project performance data indicate the program has positive
effects and has achieved its performance goals for high school completion,
postsecondary education or training, and job retention. The percentage of
Adult education students who obtained a GED or high school diploma
increased by 55 percent from 2001 to 2006."

This statement indicates that the OMB has received information from the
National Reporting System (NRS) that is reported to the Office of Education
each year and it has considered the outcomes mentioned as important evidence
for evaluating the AELS/SG as "effective." Interestingly, it does not
mention the data on learning gains that the NRS collects and reports from
the states, just the outcome data. This may mean that the issues and
controversies surrounding the measurement of learning gains, while
important to the adult education and literacy field, may not be as
important to policymakers as the less controversial measures of outcomes.

The OMB goes on to give a second line of evidence for its decision and
states: "*The program recruits, retains, and assists more people from its
target population at a lower cost than other job training programs. In
2004, the Federal cost per high school diploma or GED attained by adults or
out-of-school youth was $3,081, compared to a range of $12,000 to $90,000
for other Federal job training programs."

I'm not too certain what this means. I don't know if the OMB is comparing
the cost of a high school diploma or GED in the AELS/SG to the cost of
these same outcomes in other Federal job training programs or to the total
cost of the per student training in job training programs. For instance, in
the Job Corps a student gets both job training and education toward a GED.
Does the OMB information include the total cost of the GED and the job
training per student or has it information on the cost of just the GED
outcome? Also, I do not know where the OMB got the figure of $3,081 per
high school diploma or GED for the AELS/SG. I have figured the AELS
combined state and Federal funds per enrollee at around $800, so I don't
know where the $3,081 figure comes from. But whatever the source, it is
clear that the OMB is using a form of "cost-effective" comparison among
Federally funded programs in making its determination of the effectiveness
of the AELS/SG. This could lead to an argument for reducing funding in
other programs and directing them into the more cost-effective AELS/SG! But
this has not occurred up to now.

The third line of evidence that the OMB gives for its decision that the
AELS/SG is "effective" states: "*The program developed data quality
standards, encouraged States to use common assessment tools, and improved
the National Reporting System to collect comparable grantee performance
data."

Again, this indicates that the actions of the NRS played a major factor in
the OMB decision. The NRS approach to program improvement can be contrasted
to the approach of "scientific research" for providing evidence for program
improvement.  The NRS approach is a form of "engineering" in contrast to
experimental, scientific research comparing alternative models of programs.
The engineering approach is to develop a set of outcomes that are desired,
put a prototype program in place to pursue those outcomes, then make
successive adjustments to the system until the desired outcomes are
achieved, and then monitor output and make corrections to the program as
indicated by changes in the outcomes. This is an "optimizing" approach
which Department of Defense research over thirty years ago indicated is
more successful than experimental designs in improving instructional
programs (Bialek, Taylor, & Hauke, 1973).

This suggests that one way research in adult literacy education might
proceed is to have researchers work with practitioners to take an existing
program that is having difficulties achieving certain outcomes and then
work with the practitioners, students, administrators, and any others
involved to modify the program to achieve the desired outcomes.
Notwithstanding many problems with the NRS approach that I have criticized
in the past, the contribution of the NRS "instructional system engineering"
approach in bringing about the rating of "effective" by the policymakers in
the executive branch of the Federal government supports this suggestion for
further research using "instructional engineering."

Reference

Bialek, H. M., Taylor, J. E. & Hauke, R, N. (1973, April). Instructional
strategies for training men of high and low aptitude. HumRRO Technical
Report 73-10. Alexandria, VA: Human Resources Research Organization.

Thomas G. Sticht
International Consultant in Adult Education
2062 Valley View Blvd.
El Cajon, CA 92019-2059
Tel/fax: (619) 444-9133
Email: tsticht at aznet.net