This article was published in our January 2013 Newsletter, by David Smith, EMSTesting.com Customer Advocate
Q: Why can’t my students seem to get “Exceptional” rankings in all of their computer adaptive testing categories?
A: The students “Overall Indication” is based on 3 different criteria: Their percentile ranking, their average score, and their highest difficulty.
If they can rank above the 51st percentile when compared to all other students at their provider level, then they will be given a “Pass”. We do update these with actual numbers quite often, so it is not unusual for a student to be ranked in the 51st percentile on Friday and then find they’re at the 49th percentile on Monday.
Their average score is based on what percentage of questions they are getting correct. Instructors can see a combined summary of the students’ averages across all tests, while the student can see the combined average as well as their average score on each individual adaptive test. Students get a “Pass” in this area if they can correctly answer 65% or more of their questions in any module they are testing on.
The highest difficulty is a gauge of the most difficult question in a given module that the student answered correctly A student correctly answering questions on an adaptive test will receive more difficult questions until they get one wrong, when it will then present them with an easier question. If they get that correct they will get a more difficult question, if not they will receive a still easier question. If students can answer enough consecutive questions correctly they will hit a question with a difficulty higher than .60(meaning 60% of the selected student population do not get this question correct). Successfully answering a question higher than a .60 difficulty will give the student a “Pass” for the module.
If a student can achieve a “Pass” on all 3 criteria they will be ranked as “Exceptional” for that module. However not every module has questions higher than a .60 difficulty for 2 reasons. One is that it can be a challenge, for example, to craft an airway question that 60% of students will get incorrect that is a fair and valid question.
We’re not trying to test a student’s ability to decipher a tricky question, we’re testing their knowledge of the topic. The other is that our difficulty numbers are based on over 12 million responses to the questions from actual students. We update these numbers regularly, so this week a question may be at a .61 difficulty but next week it may have slipped down to a .59 based on so many more students answering it correctly.
Our belief is that if a student can get an “Exceptional” overall indication in at least 3 modules and a “Good” ranking in the rest that they are prepared for the National Registry and will pass on their first attempt if taken in a timely fashion.