Q: How does Platinum determine cut scores for questions and tests? (EMSTesting.com)
This article was previously published in our May 2013 newsletter by David Smith, EMSTesting.com Customer Advocate
Q: How does Platinum determine cut scores for questions and tests?
A: We use the Nedelsky method for determining cut scores.
This is primarily determined by looking at how many plausible distractors a question has. The more tempting but incorrect answers a question has, the lower the cut score. This is described a bit in the instructor guide.
The cut score of the question also plays into the difficulty. An example question could be:
Name the longest bone in the leg:
b) Hand bone
This question has one correct answer and 3 non-attractive distractors. The student should be able to rule out all of the incorrect answers pretty easily. This question would likely have a cut score of .90
Let’s set the question up differently:
Name the longest bone in the arm:
There is still only 1 correct answer, but the other 3 possible answers are also long bones in the body (2 in the arm, 1 in the leg). Because students may have difficulty eliminating any of the incorrect answers, the question might have a cut score of .36, meaning we might expect only 36% of students to get it correct and the other 64% to answer incorrectly.
We determine how many “plausible but incorrect” answers there are when creating questions, but also review the actual data of student responses. If it turns out that a very low percentage of students in the last example choose “Femur” then it is most likely not a good distractor. This may cause us to change the cut score form a .36 to a .45(meaning 1 correct answer, 2 attractive incorrect answers, and 1 non-attractive incorrect answer) . And we actively review our over 17 million unique student responses to questions to make sure we are displaying cut scores and perceived difficulty accurately.
As there are only 4 possible cut scores for a question in the program, they break down as such:
.90 cut score – 1 correct answer, 3 non-attractive distractors
.60 cut score – 1 correct answer, 2 non-attractive distractors, 1 attractive distractor
.45 cut score – 1 correct answer, 1 non-attractive distractor, 2 attractive distractors
.36 cut score – 1 correct answer, 3 attractive distractors
We then average together the cut scores of each question to give you the overall average cut score for the test.