NLA Discussion: Measuring outcomes and/or impact

[Robert P Bickerton]

     JoAnn, Taylor, et al,
     
     JoAnn raised concerns that funding sources can be inattentive to the 
     kinds of results/outcomes that are most important to our students.  I 
     believe many are, in fact, very interested in such results;  in 
     Massachusetts we are trying to make the correlation between student 
     articulated goals and results achieved the cornerstone of our 
     accountability efforts.
     
     Taylor expresses an inclination toward student articulated goals, but 
     also wonders whether they will match the macro concerns of policy 
     leaders.  In my experience (and largely in our experience in Mass.), I 
     find that the vast majority of student articulated goals are a 
     wonderful match with the educational, family, community and economic 
     priorities of policy and legislative leaders -- it's our challenge to 
     find the ways to clearly and concisely articulate the relationships.  
     
     On the other hand, I'm less convinced that we've solved the 
     "statistically sound" question since there are some who will question 
     the reliability of self-reported data.  In some cases we can establish 
     links between databases to get very hard data/results (e.g., for "got 
     a job"), but this can be intrusive on our students (most of whom are 
     not receiving a cash benefit which is often used to justify such 
     intrusive data collection) and no databases exist to verify the vast 
     majority of other goals our students strive to achieve.  I believe 
     that not too far in the future, we may need to become adept at 
     structured sampling approaches in order to verify the reliability of 
     self-reported data -- a challenge in that we need to dedicate some, 
     but not too many resources to pull something like this off.
     
     take care,
     bob bickerton
     rbickerton at doe.mass.edu


______________________________ Reply Separator _________________________________
Subject: NLA Discussion: Measuring outcomes and/or impact
Author:  nla at europe.std.com at Internet
Date:    4/18/97 4:37 AM


     
In response to JoAnn Martin--
     
Thank you for your work in this field. I regret that you are facing an 
uncertain future. Your comments make me wonder about a couple of things:
     
First, I see a real disconnect between what is statistically sound from a 
research perspective and the kind of results that a funder would want to see. 
While your model provides a simple statistical checklist that can be 
quantified, what does it tell me (as a policy-maker with a predetermined 
objective) about the impact of your work on the "larger world" I govern? If 
person A came in with Goal 1 and achieved Goal 1, then you have statistical 
proof of the effectiveness of your program. But a policy-maker will still 
ask, "Does that mean they got off welfare, their children learned to read, 
etc?"
     
Secondly, I see the issue of "learners meeting objectives" as being a moving 
target. This is not a bad thing and I would argue is a good thing. Learners 
come in for a very specific need in response to an immediate challenge. If we 
are successful, they realize that lots of other possiblities exist and their 
goal changes, grows, etc. What might have been deemed "success" when they 
entered the program suddenly seems minor compared to what they are now 
capable of. Also, learners may move on because they are satisfied for now, 
but does that mean that they are "successful" by other standards?  Was I 
"educated" when I graduated from high school? undergrad college? grad school? 
real life experience? Never ...?
     
Taylor L. Willingham
The Reading Program
Santa Clara County Library