[Bestplus] Responses from Sarah Young at CAL on our recent BEST Plus scoring questions

[Schwerdtfeger, Jane]

Hi, everyone--I'd like to introduce Sarah Young, CAL's adult ESL Content
Specialist.  She will be joining our list discussions, and I hope this will
allow us to get her feedback as we bring issues up, rather than have a bit
of a lag time between the great discussions generated here and then getting
a response.  Welcome, Sarah!  

Below are Sarah's responses to the questions we were recently discussing on
the list.  She also addressed some of the questions that came up in our
September 16th refresher, which I hope will help to clarify some of the
confusion many of us felt.  Thank you, Sarah, for your thoughts.  

Jane Schwerdtfeger

Dear Massachusetts BEST Plus Testers: 
Breana Abbott and Frank Finamore passed on some questions to me that you had
regarding scoring the BEST Plus, and I'm pleased to be able to discuss our
answers with you. Before answering these questions, I'd just like to
introduce myself. I've recently begun working on the BEST Plus team as an
adult ESL content specialist. I also teach part-time at the REEP adult ESL
program in Arlington, VA and as a test administrator there, I'm aware of
these types of issues. I see a few questions here that I'd like to provide a
response to, or in some cases simply confirm what you all have been
discussing. 
1) Repeating a word vs. an entire question and related scoring issues: 
What Iris said below is correct. When it's necessary to repeat the entire
question, if the examinee either misunderstood the question, asks for a
repeat, and just gives you a blank stare, the highest score that could be
assigned for LC is a 1. 
>Hi all,
>
>My practice always has been that if the student repeats a word with
>questioning intonation to check understanding, and the word is correct,
>I simply nod or say yes, then wait for an answer (giving a 2 for LC if
>the answer is appropriate). It wasn't my understanding that the word
>needs to be repeated by the administrator. 
>
>However, if the student repeats a word to check understanding and the
>word is not correct, then I repeat the whole question one time. Then,
>of course, the highest score that could be assigned for LC is a 1. 
>
>What have you all been doing?
>
>Iris
2) Communication scoring: Difference between a 2 and a 3: 
Thanks to everyone for taking the time to really understand the rubric and
apply it accurately to real-life testing situations. Many times, these
authentic testing situations can provide some cause for confusion as we try
to determine if an examinee is "sporadically difficult to understand" (but
we eventually get the point) or clearly "comprehensible and easy to
understand." This can be a sticky point, especially because as
teachers-turned-testers, we try hard to monitor ourselves and not fill in or
overcompensate for what we *think* the examinee is saying. I've been through
two BEST Plus trainings and I remember both trainers emphasizing this point.

That said, if we return to the rubric, a "3" score stipulates that "Despite
inaccuracies, the listener does not need to fill in to understand meaning."
If, at the end of an examinee's response you have reasonably understood him,
even if some of the language used along the way was problematic (which is
reflected in the language complexity score), you should feel very
comfortable in giving him a 3. However, if the examinee's complete response
leaves you feeling like you have not understood the entire idea, or that
inaccurate words have been used that truly impede communication, you should
score a 2. (I can think of one example where I thought a student was talking
about the U.S., but as the response went on, I realized that he was talking
about his own country, which, due to the topic, was completely confusing to
me. I understood the literal words he was saying, but he did not clearly
communicate the actual meaning of his discourse.) 
In responding to Betty's message below (Oct. 25), you do want to be
conscious of scoring the whole response ("not simply the final
result/answer, which might ultimately be perfectly clear and well
communicated"), but ultimately, what does matter is how well the examinee
communicated the meaning overall. 
3) Codeswitching/using native language or native pronunciation in responses:

A few examples have been brought up about how to score for Communication
when Spanish, for example, is in the response. Although it would be possible
to try to analyze each word in a response to figure out just how much
meaning was communicated, it has not been my experience that that is
actually how the scoring rubric works. We here at CAL know that this can be
a difficult point, and I appreciate efforts to come up with a final,
objective response so that everyone is on the same page. When an examinee
codeswitches or uses an L1 word, the tester should first respond by saying,
"In English, please." If they are able to rephrase the response in English,
then no harm done -- but we do want to give them that reminder/chance to
give an English response. Many students, due to nerves or whatever, don't
realize that they've even switched to their L1. If, however, they are still
unable to give you an all-English answer, you have to score the response
as-is. Score the response according to how much you understood, and in the
context of your local situation (i.e., if something is pronounced in a
Spanish accent, but it is something that you are familiar with in your
community). If you get the general idea of what is being said, even if a
word or two is in Spanish (many of my examinees use connectors such as
"pero" or "pues" when responding to questions, probably subconsciously),
then you can score a "3," according to what the rubric says. 
4) Zeroing out: 
The question was brought up by Janet (9/22/05 ) about if it is possible to
give a zero in Language Complexity/Communication following a 1 or 2 in
Comprehension. For all practical purposes, it really is not. The examinee
might nod their head or respond in the L1 to indicate their understanding of
the question, but you can't score a nonverbal or non-English response.
Again, in this case it is worthwhile to prompt the student to answer
verbally and in English. 
Please let me know if there are other questions you'd like me to address. As
an adult ESL instructor/tester/content person, I'm always interested in
these types of discussions. 
Sincerely, 
Sarah Young Center for Applied Linguistics 4646 40th St. NW Washington, DC
20016 Phone: (202) 362-0700 ext. 529 Fax: (202) 362-3740 Web: www.cal.org
<http://www.cal.org/> Email: sarah at cal.org <mailto:sarah at cal.org> CAL:
"Improving communication through better understanding of language and
culture" 
			Thank you to everyone. Thanks, Iris, for your clear
write up of the scenario. 
			Betty
			P.S. As I was testing this morning (actually
co-testing with another assessor), we began talking about Communication
scores.
			For the first time I articulated that in scoring
that part, I want to be conscious of scoring the whole response, not simply
the final result/answer, which might ultimately be perfectly clear and well
communicated. 
			This is kind of a "duh" moment, but as a trainer and
resource for my own assessment team, it's another way to help people be
aware of the difference between a 3 and a 2 in Communication. The fuzziness
or confusion along the way is what the "sporadically, generally, difficult"
are all about, even if you finally do understand what the student
means/says. 
			What do you think? 
			Perhaps I'm just in "thinking too much" mode
again....These are all the interesting conversations that come out of a few
co-tests as we (belatedly) get our testing in gear.
			P.P.S.S. See many of you tomorrow at NETWORK, bjs
			Betty J. Stone
			ESOL Program Administrator
			SCALE - Somerville Center for Adult Learning
Experiences
			167 Holland Street
			Somerville, MA 02144
			Tel: 617-625-6600, Ext. 6933
			FAX: 617-623-8528
			Email: bstone at K12.somerville.ma.us
<mailto:bstone at K12.somerville.ma.us>