[Bestplus] BP meeting follow up

[Lubets, Stacey]

Dear Fellow BEST Plus Trainers,
Hi all! I hope that everyone is well and enjoying the new year. I realize
it's been a while since our September 16th BEST Plus refresher training, but
I do want to follow up with you all and give you an update on where we are
now. Jane and I really appreciate all the great feedback you provided at the
training. It was particularly valuable that the staff from CAL was there to
get your input as trainers and to address some of the concerns that were
brought up. Thanks so much for your patience as we work toward resolving
some of these concerns. In recent discussions with the folks at CAL, we were
able to hash out some of the key issues that were raised at the September
training. The following items were included in our discussions:
Can a review of essential grammatical structures be a part of the training?
We have heard from some trainers that trainees may not be able to identify
some of the grammatical structures they'll need to listen for when scoring a
learner. Sometimes trainees may be reluctant to admit in front of a group
that they do not know what a subordinate clause is, for example. This is a
tricky issue; on the one hand, we want to make sure that participants have
the resources they need to fully understand the rubric, including a full
understanding of grammatical structures such as adverbial clauses and
subordinate clauses, to interpret the rubric appropriately. It is very
important, however, that we do not give trainees any sense that grammar is
more important than other aspects of Language Complexity, or more important
than Listening Comprehension or Communication, in interpreting the BEST Plus
rubric. The following is how CAL and ACLS hope to strike a balance in
addressing this question.
Staff at CAL and ACLS agreed that trainers can touch on grammatical terms
referred to in the BEST Plus rubric in Section IV (Introduction to Scoring),
during the 10 minutes that participants are going over the BEST Plus scoring
rubric, before they view the video and go over the benchmarks. Once you have
introduced the Language Complexity category, we suggest trainers say that
you don't want to dwell on grammatical aspects of Language Complexity, but
that if trainees have any questions about some of the grammatical terms
referred to in the rubric, they can refer to a few resources. 
· One is the Definition of Grammar terms, from page E-2 in the February 2005
Trainer's Manual. (I have attached a sheet here for you; CAL asked that we
remove the sentences addressing trainers at the beginning and the end, so I
have done that on this sheet. Please do not hand out a Xerox of the page
from the Trainer's Manual). You can draw from this to offer trainees some
definitions and examples of grammar terms used in the rubric. 
· Two, please refer trainees to the BEST Plus Language Complexity Reference
Sheet (yellow paper), currently included in the front of the TAG, to provide
further clarity and examples in this area. This sheet can be referred to by
participants both during the training session and prior to administering
tests. 
If trainees have any questions, you can briefly answer them here, but we do
not want to spend more than a couple minutes on grammar here. Tell trainees
if they have additional questions, they can ask you at a break, or during
lunch, or after the training. 
Please also tell trainees that when they view the benchmark video, they see
examples of how the types of grammatical structures are integrated into a
score of a "2" on Language Complexity, or in a score of "3" in Language
Complexity. 
Here are some additional suggestions if you are looking for additional ways
to "beef up" trainees' understanding of the grammatical structures
referenced in the BEST Plus rubric: 
· To reinforce understanding of how the use of grammatical structures fits
into evaluation of a response using the BEST Plus Scoring Rubric, have
participants underline or highlight specific grammatical structures within
the text of the responses shown on the training video (transcripts in the
TAG) that qualify a response as a 2 or 3 in Language Complexity. 
· Prepare and have in mind a few key points to share with participants who
have questions about the grammar during breaks, lunch, etc. - for example,
reminding them that grammar structures are not the only basis for scoring
Language Complexity; that elaboration, detail, and variety are also
important; and that they should have confidence in their understanding of
these grammar points from their prior teaching experiences. You might
suggest that they take some time on their own to just listen to native and
non-native English speakers and practice noticing, in a holistic way, what
makes their language more or less complex. This holistic sense of language
proficiency may help calm their fears about their understanding of just one
aspect (grammar) of the Language Complexity score. 
· Suggest that once test administrators are back at their program sites they
engage in discussions with other teachers and test administrators about all
aspects of the rubric; these types of professional discussions often enhance
or clarify our own understandings of the complex linguistic system referred
to as "language proficiency."
We don't want to assume that test administrators don't understand these
grammar terms, but we also want to make sure trainees understand the rubric
as fully as possible when they leave the training. We also want the use of
BEST Plus to not only be an effective tool for assessment and reporting
purposes, but also to provide opportunities for teachers and test
administrators to be engaged in their own professional development, by
thinking critically about language proficiency and the linguistic
characteristics of their adult ESL learners. 
Can the cooperative "jigsaw" activity be shortened or omitted? 
According to CAL, this is not something that can be shortened. Their
response states, "The BEST Plus training workshop was designed to prepare
test administrators to use BEST Plus by ensuring that they understand
correct test administration and scoring procedures, and that they score
accurately. The workshop also aims to educate participants about testing in
general, about the development of BEST Plus, and about how BEST Plus fits
into the testing world. The Cooperative Reading Activity is the only
activity during the training, which highlights important information about
the appropriate use of the test as well as practical information about the
computer and print versions. This activity should not be shortened or
omitted. However, there are some ways that trainers can help this activity
flow more smoothly. First of all, make very clear in the directions exactly
how the jigsaw will be accomplished. Then, participants should scan the
material fairly quickly in order to pick out the information needed to
answer the questions. The small groups should then share the information.
Finally, as a large group, only debrief on the questions highlighted in the
Trainer Manual as ones particularly important to note." 
In keeping with CAL's response on how to keep this cooperative learning
activity moving along, here are a few more suggestions from ACLS: First,
preface the activity by stressing that this is a highly efficient way to
learn some of the key points that a BEST Plus administrator needs to know.
This gives participants a more positive spin on why this activity is
included in the training. Another area that can be expedited is the
pairing/grouping process. It's quick and simple to pair people with their
neighbors so they don't have to relocate. Also, it may help to circulate and
watch that each group is on task, answering any questions as you go around
the room. 
Is it possible to drop the print-based section? 
Both CAL and ACLS agreed that the amount of time dedicated to the
print-based materials can be minimized by simply mentioning that there is a
print-based version of BEST Plus, although Massachusetts does not use it. It
is important that people know this because BEST Plus administrators from all
over the country call CAL with questions. CAL's support staff may ask
callers which version they have, the print-based or the computer-based, in
order to know how to answer their question. This can cause some confusion if
callers are not aware that both versions exist. 
It seems that there's something about the last hour of the workshop that
makes the participants get 'scoring fatigue'. Would it be possible to do the
practice tests earlier, so people see the test and touch the computer early
on? 
CAL has determined that the practice component is designed so that
participants have a chance to synthesize and apply everything that they have
learned during the day. This synthesis cannot happen if the practice is
moved before any of the other activities. The full demonstration of the
test-both on the video and the in-person demo the trainer conducts-give
participants a chance to see the test early on. Therefore, CAL requests and
ACLS agrees that we keep the structure and timing of the training as is. 
Do we need to present all of the slides on reliability and development? 
CAL believes that the reliability and test development slides are essential
components of the training. Providing this background to participants,
although somewhat technical in nature, legitimizes the claims that BEST Plus
makes about use, consistent scoring, and input from the field. It also
clearly demonstrates that it is possible even for novice test
administrators, such as the participants themselves, to consistently score
examinees. The reliability slides provide the foundation for the final two
slides in the section that feature important graphics (convergence graph and
NRS/SPL/BEST Plus alignment). The information presented in this section also
contributes to participants' professional development, both as ESL
instructors and test administrators. 
>From the ACLS perspective, we do understand your desire to make the
PowerPoint portion of the training less technical in nature. We concluded
that trainers should try to make this section more interactive by asking
participants to share what they already know about test properties and
development (and their importance in assessment, which is ultimately linked
to instruction), or by using an analogy, some examples of other uses, or
needs for reliability in other aspects of their lives. It may also help to
draw from your own experiences to keep the PowerPoint portion lively and to
connect some of the concepts to real testing situations. Also, some slides
can be confusing and these points can help to simplify your explanation. I
am attaching some talking points that I personally use during my trainings
to help alleviate the "glazing over effect" trainees sometimes exhibit
during the PowerPoint presentation. 
What can we do about observing trainees and determining if they're approved
during the training workshop? How can we be sure that a test administrator
at a training workshop fully understands all aspects of BEST Plus and is
ready to administer? 
This is something that we are continually trying to address. In addition to
our remediation process, which I am attaching for your review, CAL has some
suggestions to help ensure that test administrators are well prepared to
administer BEST Plus:
· Have new test administrators co-test with a more experienced test
administrator for some of their 10 practice tests in order to be approved-
if programs have the staffing available
· Compare pre-test and post-test scores done by the same test administrator
to see if they are reflecting any inconsistencies.
Trainees need final scores on their practice administrations to check their
accuracy. Is this possible?
This option is not available from CAL at this time. However, as you may
already know, CAL is in the final stages in the development of the Test
Administrator Scoring Activity toolkit to help ensure that test
administrators are administering and scoring accurately once the training
has concluded and they are back at their program sites. This toolkit will
feature a video with a refresher on BEST Plus administration and scoring,
video clips with benchmark samples of all scores on the rubric, and three
quizzes to test the participants' scoring ability. CAL was delayed in
finalizing this, but anticipates a nationwide May 2006 rollout on this
project. Although ACLS has not yet determined when the Test Administrator
Scoring Activity toolkit will be available for use in Massachusetts, we are
confident that once this is in place, scoring accuracy and test
administrator confidence will be improved for those who participate in the
refresher scoring activity. 
We would like to be able to archive records of students who have left the
school/program, so their names are no longer on the scroll list, but are
still accessible. We would like a way to edit the database to add (not
delete and correct) information (e.g., wrong date because computer didn't
keep the time and the tester didn't notice, edit class and level)?
One item that was mentioned in the training was the editing limitations of
the Score Management Software. An administrator cannot edit program
information such as class, location, etc., once this information is entered.
Therefore, if students move from one class to another, (as often happens in
ESOL), the student records cannot be updated. At this time, CAL is unable to
make any changes to the Score Management Software, but will take all
suggestions into consideration in the next iteration of the software. They
thank you for your input on this.
In the meantime, there is a way around this problem. Since the student
Registration ID field can be edited and is only used for in-house record
keeping, I found a simple method to identify what class a student is in. I
assign a class code (from SMARTT), followed by the students' first initial
last 4 digits of their last name. For example, Stacey Lubets, class code 400
is entered as ID#400slube. This way, if the student changes classes this
information can be easily edited and can be seen at a glance.
Does CAL have any plans to revise the test itself (i.e. rephrase "Tell me
more... and other parts of the script)? One example cited was the question,"
Tell me about your favorite holiday." If a student responds by saying
'Christmas,' the prompt "tell me more" may merely elicit a list of other
holidays rather than any elaboration about Christmas. 
There is not a plan for revision of the test in the immediate future;
however, CAL is compiling a pool of all your comments and suggestions
concerning test revision. They appreciate the feedback you shared with them
at the training on this matter. Test administrators are reminded that they
should score each response according to the rubric; in the example given
above, a list of other holidays is still a scorable, reasonable response and
the test will work itself out and adapt to the examinee's proficiency level
as necessary.

In summary, I hope this email clarifies some of the issues that were raised
at our BEST Plus refresher meeting. Please let me know if there is anything
that is not clear on any of the points listed above. I also encourage all of
you to submit your questions and concerns on our BEST Plus trainer listserv.
It's great to have any open forum to discuss what's on everyone's mind, and
for everyone to have access to responses. In addition, CAL now sends a
newsletter to trainers, which you may have already received. Please take a
few minutes to look this over for updates on any BEST Plus related issues. I
look forward to hearing from you soon and offering any support you may need
in your role as trainers.
Best regards,
Stacey
Stacey Lubets
Adult & Community Learning Services
350 Main Street  4th Floor
Malden, MA 02148
Phone: 781-338-3823
Fax: 781-338-3394