Test Wise Answers

Not too many guesses at the Test Wiseness questions. I guess that's ok, I didn't answer them at the seminar. I think that at some level you read the nonsense and don't want to be the one that gets it wrong and then finds out you're an idiot.

Ben did very well though.

The point of this thing was to help us develop test questions (items) that didn't give away the answers and that had good wrong answers (distractors). This little exam demonstrates some of the problems with bad items and distractors. They presented it to us at the top of the day before we started writing items.

The answers go: ABCDAB. Which in and of itself is a bad thing because it is a pattern. They told us that regardless of the letter of the right answer in the items we were writing, they would go through and change the percentages of each letter to make them equal and then totally randomize the order. So people looking for patterns will be out of luck, and someone trying the "answer with all B's" method will get a 25%.

I assume a 25% will not pass. We set that at the next meeting I think.

So, here are the clues that tip the answers:

• Question one has part of the answer (key) in the question (stem). They told us not to do that.
• Question two has only one choice that works grammatically. Dead give away. It apparently doesn't matter that "an" goes with "A."
• In question three, all the distractors are absolute and the key is conditional. We were told that it is hard to write a truly defensible multiple choice question where the key can be absolute - someone can always think of the one exception. Therefore absolute distractors are a telltale.
• Answer D to question four has more qualifications than the others. This is another give away for the same reason as above - trying to be specific enough.
• Question five has the same issue: two conditions next to distractors with one.
• Question six is a well designed question, except if it is on the same test as question two. This is called "cluing," when you can use the text of one question to answer another one.

I have to say that in the case of the last example I sometimes find myself cluing on purpose on exams I have written for my classes. It was the sort of thing I was taught to try to do when I was taking tests in school. My students notice it with surprisingly little frequency.

So that was most of the "workshop." In addition to this they also said that "all of the above" and "none of the above" were bad keys because they typically have a higher percentage of being true.

I had this Bio teacher in High School who gave multiple choice tests with 8 keys: four answers and then "two of the above," "three of the above," "all of the above", and "none of the above." I'll have to ask next time if that is a reasonable construction.

Thanks again to the people at AMP.