It’s been over a year since I first published “NBME Self-assessment Tests and USMLE Review.” Little did I know it will become the most popular of my post with over 14,000 pageviews in the past year. Since that time a lot of things have changed, hence this update.
How well does NBME predict your USMLE Score?
That has been the number one question asked of me since I wrote about this topic. My answer is still the same. Fairly well. Although correlation is never 100% more like 70 to 80%. However, certain development in the past few years have made the assessment tests less reliable for some people.
One of the reasons for this problem is that a lot of NBME questions, especially the earlier forms of step 1 have been discussed extensively in various forums. What’s more some of these posters did not even bother to warn people that what they are discussing are NBME form questions. Going into the NBME assessment tests knowing some of the answers already can invalidate the predictability of those assessment tests. In fact just knowing some of the questions beforehand can also invalidate the results. The reason is that knowing the questions ahead, means you’ve had time to consider the questions and possible answers before, not just the 1 minute or so that you will actually have in a real exam. That can skew your result.
It has also come to my attention that some NBME questions or variations of those questions have come out in online qbanks. That is bad news, too as it has the same effect of invalidating some of the questions in the NBME forms. Even if you did not get the answer, encountering the questions in the NBME form for the second time rather than the first time means you had more than the 1 minute allowed in the exam to think through the questions and look for the answers.
So what are the remedies for these problems that seem to have cropped up recently. Well first is to actively avoid discussing posted NBME questions in the forums. Of course since some posters do not have the courtesy of even warning people about it, avoid participating in any discussions on questions in forums unless you know for sure that they did not come from an NBME form.
Another solution is to try to use the later forms. In Step 1 this would be form 4. 5 and 6. NBME forms for Step 2CK does not suffer from the same problems as those in Step 1 as they are not discussed as often as those of Step 1.
Doing at least 2 forms instead of just 1 as I recommended before may also help. Just make sure one of the form is not the first 3.
What about the “Downloaded” NBME Self-assessment test version.
I can never understand the popularity of the so-called “downloaded” version of the NBME. IF saving a hundred dollars or so is worth scoring low or failing the USMLE altogether, then it is understandable. But ruining your long term career to save a couple of bucks is not a very intelligent move.
In earlier posts, I have said that the value in the NBME does not lie with the fact that the questions mimic the USMLE. In fact, in general, they are much easier than the USMLE. The main value of the NBME forms is that they are fairly reliable predictors of performance in the USMLE due to the correlation they’ve done with NBME results vs. actual USMLE performance.
However, recently, some posters in my blog have commented that they are able to “predict” their USMLE scores, since there are answer keys and correlation tables available with the downloaded version. So I decided to give it another look.
So what did I find out? Well, first, there is a problem with the answer keys. There were some answers that I completely disagree with. In some cases where I myself am not sure what the right answer is, I tried to verify the possible correct answer by researching them and I still cannot decide what the right answer is even after searching through textbooks and the internet. Therefore, there is a question of how accurate the raw score one is getting for each of those NBME forms are. And that is a major problem. In my case, anywhere from 2 to 8 answers in each form fall into this category and for me, an “unknown” of 4 to 16% in the raw scores completely shoots down any chance of actually knowing the exact raw score you should be getting.
A second major problem is that there is only one correlation table in existence, instead of the 6 correlation table I am expecting. That is one per form. There might be additional correlation tables out there, but the one I got is just 1. Why six tables? Because the forms are of different levels of difficulty, you expect the same person will get different number of questions right in the different forms depending on the level of difficulty. So you need a correlation table for each form to make them comparable. Having only one correlation table means we don’t even know to which form this correlation table belongs. See the problem, now.
A third really big problem is that people who used this downloaded forms has a tendency to keep on using them throughout their prep multiple times. Probably because it’s free. In fact, it was justified that since they did not try to look at the answer, they can redo the same form and expect it to still be accurate in predicting their scores. That is actually wrong. Again, one of the reasons why USMLE is hard is the time limit imposed in answering questions. When you go through the same question multiple times, you’ve had more than the 1 minute per question limit imposed by USMLE to think of the answer and therefore will tend to score higher. That skews the predictability of the NBME.
Therefore, again, do yourself a favor and use only the online NBME assessment tests and use them only when you feel you are ready for the USMLE, to confirm your readiness. The “downloaded” NBME forms may seem free, but it’s hidden costs may be greater than you are willing to pay.
USMLE World Assessment Tests
I’ve had more time to gather information about this relatively new resource. So far based on talking to my students, posters in my blogs, people who have emailed me and reading various forums, my conclusion is that the UW assessment tests is just as good as the NBME assessment tests, so far. Although there have been some observations that UW tend to be overestimate your scores in comparison to NBME, this does not seem to happen in all cases and the score difference is not too big. So all in all, I believe the UW assessment tests have enough track record by this time that we can safely say, they are fairly accurate in predicting USMLE scores. But as in all assessment tests, correlation is never 100%, therefore expect some deviation from predicted scores in the final result.