[Bestplus] FW: BEST Plus question

[Schwerdtfeger, Jane]

Hi, everyone, we thought you all would want to read this email exchange and
question about scoring BEST Plus.
Jane

-----Original Message-----
From: Schwerdtfeger, Jane 
Sent: Thursday, July 13, 2006 3:21 PM
To: 'Sarah Young'; Schwerdtfeger, Jane; Carey Reid; Lubets, Stacey
Cc: richard.rakobane at umb.edu; aespindo at bristol.mass.edu;
creid at worlded.org; dmccormack at hcc.mass.edu; jfischer at necc.mass.edu;
mfreeman at qcc.mass.edu
Subject: RE: BEST Plus question


Sarah, thank so much for your response--this is helpful.  CACs, this is also
helpful information to clarify for other practitioners.  Lee, thanks for
asking the question--

Jane

-----Original Message-----
From: Sarah Young [mailto:sarah at cal.org]
Sent: Monday, July 10, 2006 12:09 PM
To: Schwerdtfeger, Jane; Carey Reid; Lubets, Stacey
Cc: richard.rakobane at umb.edu
Subject: RE: BEST Plus question


Hi Jane and all,

Interesting questions.. Partial understanding can be tricky for some
people to score correctly, especially because as ESL teachers and
advocates, we want to give our students (examinees) the benefit of the
doubt whenever possible. However, I agree with Lee - according to the
transcript of what was said in the questions and answers, there really
isn't enough understanding of the question on the part of the examinee
to indicate even partial understanding. Simply repeating a key word in
the answer is not enough to demonstrate understanding of the underlying
question. My thoughts echo Lee's rationales behind each of the 0 scores
in Listening Comprehension. This is also reflected in the language used
in the Test Administrator Guide, Section II-2: "Partial understanding is
awarded when the majority of the setup and question are understood: for
example, the response is to an embedded question within the test item or
only one word in the question is misunderstood." As you can see,
trainers should not be telling test administrators that it is partial
understanding if one word is understood - but rather the opposite.

I hope this is helpful - but Lee already said things quite well!
Thanks,
Sarah Young
Center for Applied Linguistics
4646 40th St. NW
Washington, DC 20016
 
Phone: (202) 362-0700 ext. 529
Fax: (202) 362-3740
Web: www.cal.org 
Email: sarah at cal.org
CAL: "Improving communication through better understanding of language
and culture"
   

-----Original Message-----
From: Schwerdtfeger, Jane [mailto:JaneS at doe.mass.edu] 
Sent: Wednesday, July 05, 2006 4:53 PM
To: 'Carey Reid'; Schwerdtfeger, Jane; Lubets, Stacey
Cc: Sarah Young; 'richard.rakobane at umb.edu'
Subject: RE: BEST Plus question

Lee, I think you are right in how you interpret these scores, and I
believe it is incorrect to score for partial understanding if the
learner understands one word of the question. I am ccing Sarah Young at
CAL, but that is my take. If this is correct, we do not want scorers
thinking (and
scoring) otherwise.

Sarah, can you respond to make sure we are on the right track, before we
send it to our BEST Plus list serve?  (Thanks so much for sending this
question, Carey, it's a good one.) Thanks, Jane

-----Original Message-----
From: Carey Reid [mailto:Carey_Reid at worlded.org]
Sent: Friday, June 30, 2006 10:08 AM
To: JaneS at doe.mass.edu; slubets at doe.mass.edu
Subject: Fwd: BEST Plus question


Hi Jane,

I told Lee that I would forward this question to you first rather than
post it on the B+ listserv because it might lead to policy and need
discussion with CAL.  If this is your preference, you can post the
question and your response to the Bestplus at lists.literacytent.org when
you're read.  The listserv is still operational.

Carey.

>>> "Lee Haller" <l.haller at jpccalp.org> 6/29/06 7:34 PM >>>
Hi Carey.  Could you please send this BEST Plus question to the list?
Thanks, Lee

I apologize if this issue has been discussed on this list before.  I
don't have access to the past discussions because I changed email
addresses a year ago.

We did a BEST Plus recalibration a couple of weeks ago where we had
students come in for practice tests.  There were three teachers in each
group:
Teacher A administered a practice test to the student, asking questions
and scoring normally Teacher B observed the test, wrote how she would
have scored each response on a paper, then after scoring each response
looked at the computer and recorded teacher A's scores Teacher C took
notes on the questions and also scored.

I mention how we did this because it was a very successful training. 
I
understand we got the idea from SCALE.  Thanks, Betty!

A question came up in our scoring differences.  The question was about
when to give partial credit for listening comprehension, and when to
give a 0.
Here are a couple of examples (I don't remember the exact wording of the
questions--this is from my notes):

Q1:  Do you think airplanes are safe or dangerous?
A1:  Yes, I do airplane

Q2: What are some things you and people in your neighborhood can do to
make it a better place to live?
A2:  My neighborhood is nice.

Q3: When I was looking for a job I looked in the newspaper.  How did you
find your job?
A3:  Repeat please
Q: (repeated)
A3:  My job is independent.

In all three cases, the other people in my group said they had been told
in the training that if a student shows they understand one word of a
question, that is partial understanding and they should get a 1 in LC
for that response.  I remembered the training as being that partial
understanding meant partial, like they understood about half of the
idea, more than just one word.  I scored all 3 as zero in LC.

For Q1, I thought the response was inappropriate and did not show
understanding of anything other than the word "airplane".
For Q2, I thought the response did not show even partial understanding
of this complex question, and only showed understanding of the word
"neighborhood".
For Q3, I thought that even after repetition, the student's response did
not show understanding of anything other than the word "job".  It is
possible that the student meant she was self-employed so didn't have to
look for the job, but this is a real stretch.

Each question was part of a series of questions, of course, where the
student was able to understand and successfully answer simpler questions
about airplanes, neighborhoods, and jobs.  To me, the reason to have a
series of questions on one theme is to differentiate students'
understanding
of questions which are simpler or more difficult relative to each other.
If a student catches on to the general theme of the thread and then just
says a sentence with the word "job" or "neighborhood" in response to the
one word she understood, that does not show even partial understanding
of the question asked.  It does show partial understanding of the
thread, but that's not what we're measuring.

I discussed Qs1 and 2 afterwards with Richard Rakobane and he disagreed
with me about Q1 but agreed about Q2--essentially, he thought that with
a simple question, one-word understanding could be partial
understanding, but with a more complex question, one word is not enough.
Since he and I disagreed, I decided to submit the situation to the
group.

Thanks.

Lee Haller