We previously tackled one common myth about the USMLE. (See “Is the version of the USMLE for IMGs harder than for AMGs?” ) Now we will tackle another one. In order to answer this question we need to understand how the USMLE come up with questions to include in the USMLE.
The USMLE states that in order to pass the exam, one should answer correctly between 55% to 65% of the questions presented in the exam. The reason a range is given is that different exam sets have different difficulty levels and the more difficult the exam set you get the lower the number of correct answer you actually need to pass. The difficulty level of an exam set depends on the number of difficult and easy questions inside the exam set. The more difficult questions there are, the harder the exam set will be and the lower the percentage correct you need to pass the exam. However, there is a limit to how hard or how easy the exam should be since there is a limit to how high or how low the percentage of correct answers needed to pass the exam.
How does the USMLE determine how hard a question is?
Well if you think they appointed a panel or guru to decide how hard the exam is, you are wrong. They are very fair about it. The questions are given to actual exam takers and if too many get it wrong, it is considered too tough and is removed. Ditto if too many get it right. That is the basis for experimental questions in the actual exam. Contrary to popular myth, experimental questions are not tough questions. They could be tough or easy. What they share in common is that the USMLE do not have enough data to decide whether it is too tough, too easy or just right for the USMLE. That is also the reason that it is not included in your score.
How does the USMLE determine if the question is just right for the exam?
Well the mandate of the USMLE is to come up with an exam that 90% of AMGs will pass and 10% will fail. That is the norm used by the USMLE. More than 90% pass, it’s too easy less than 90% pass it’s too tough. Recently, the USMLE raised the passing grade for the USMLE since close to 94% of AMGs are passing the exam.
So is the exam getting tougher?
Theoretically, it can. As the ability of AMGs to answer tough questions get better, more and more tough experimental questions get incorporated eventually into the actual examination. And as shown by the recent increase in the passing score, that seems to be the case. But we should also remember that at most, 10% of the questions in a given year are experimental. And if we posit that not all questions will eventually be incorporated to the exam(some will be too hard or too easy), then at most 5% of the exam will change per year. (though I suspect it is less than that, since it’s not easy to come up with questions) In fact, AMGs were getting better at answering tough questions faster than tougher questions can be incorporated into the exam pool that the USMLE had to increase the passing score.
So how much you really know is not the basis for passing the USMLE but how much you know in relation to everyone else especially AMGs. So it does not really matter if the USMLE is getting tougher every year, if it does it means everybody is getting better every year and you should too.