It’s been over a year since I first published “NBME Self-assessment Tests and USMLE Review.” Little did I know it will become the most popular of my post with over 14,000 pageviews in the past year. Since that time a lot of things have changed, hence this update.
If you have not read the previous two posts on this topic, please do so as I will not be repeating what I have said there in this post. You can access Part 1 here and Part 2 here.
How well does NBME predict your USMLE Score?
That has been the number one question asked of me since I wrote about this topic. My answer is still the same. Fairly well. Although correlation is never 100% more like 70 to 80%. However, certain development in the past few years have made the assessment tests less reliable for some people.
One of the reasons for this problem is that a lot of NBME questions, especially the earlier forms of step 1 have been discussed extensively in various forums. What’s more some of these posters did not even bother to warn people that what they are discussing are NBME form questions. Going into the NBME assessment tests knowing some of the answers already can invalidate the predictability of those assessment tests. In fact just knowing some of the questions beforehand can also invalidate the results. The reason is that knowing the questions ahead, means you’ve had time to consider the questions and possible answers before, not just the 1 minute or so that you will actually have in a real exam. That can skew your result.
It has also come to my attention that some NBME questions or variations of those questions have come out in online qbanks. That is bad news, too as it has the same effect of invalidating some of the questions in the NBME forms. Even if you did not get the answer, encountering the questions in the NBME form for the second time rather than the first time means you had more than the 1 minute allowed in the exam to think through the questions and look for the answers.
So what are the remedies for these problems that seem to have cropped up recently. Well first is to actively avoid discussing posted NBME questions in the forums. Of course since some posters do not have the courtesy of even warning people about it, avoid participating in any discussions on questions in forums unless you know for sure that they did not come from an NBME form.
Another solution is to try to use the later forms. In Step 1 this would be form 4. 5 and 6. NBME forms for Step 2CK does not suffer from the same problems as those in Step 1 as they are not discussed as often as those of Step 1.
Doing at least 2 forms instead of just 1 as I recommended before may also help. Just make sure one of the form is not the first 3.
“Downloaded” NBME version.
I can never understand the popularity of the so-called “downloaded” version of the NBME. IF saving a hundred dollars or so is worth scoring low or failing the USMLE altogether, then it is understandable. But ruining your long term career to save a couple of bucks is not a very intelligent move.
In earlier posts, I have said that the value in the NBME does not lie with the fact that the questions mimic the USMLE. In fact, in general, they are much easier than the USMLE. The main value of the NBME forms is that they are fairly reliable predictors of performance in the USMLE due to the correlation they’ve done with NBME results vs. actual USMLE performance.
However, recently, some posters in my blog have commented that they are able to “predict” their USMLE scores, since there are answer keys and correlation tables available with the downloaded version. So I decided to give it another look.
So what did I find out? Well, first, there is a problem with the answer keys. There were some answers that I completely disagree with. In some cases where I myself am not sure what the right answer is, I tried to verify the possible correct answer by researching them and I still cannot decide what the right answer is even after searching through textbooks and the internet. Therefore, there is a question of how accurate the raw score one is getting for each of those NBME forms are. And that is a major problem. In my case, anywhere from 2 to 8 answers in each form fall into this category and for me, an “unknown” of 4 to 16% in the raw scores completely shoots down any chance of actually knowing the exact raw score you should be getting.
A second major problem is that there is only one correlation table in existence, instead of the 6 correlation table I am expecting. That is one per form. There might be additional correlation tables out there, but the one I got is just 1. Why six tables? Because the forms are of different levels of difficulty, you expect the same person will get different number of questions right in the different forms depending on the level of difficulty. So you need a correlation table for each form to make them comparable. Having only one correlation table means we don’t even know to which form this correlation table belongs. See the problem, now.
A third really big problem is that people who used this downloaded forms has a tendency to keep on using them throughout their prep multiple times. Probably because it’s free. In fact, it was justified that since they did not try to look at the answer, they can redo the same form and expect it to still be accurate in predicting their scores. That is actually wrong. Again, one of the reasons why USMLE is hard is the time limit imposed in answering questions. When you go through the same question multiple times, you’ve had more than the 1 minute per question limit imposed by USMLE to think of the answer and therefore will tend to score higher. That skews the predictability of the NBME.
Therefore, again, do yourself a favor and use only the online NBME assessment tests and use them only when you feel you are ready for the USMLE, to confirm your readiness. The “downloaded” NBME forms may seem free, but it’s hidden costs may be greater than you are willing to pay.
USMLE World Assessment Tests
I’ve had more time to gather information about this relatively new resource. So far based on talking to my students, posters in my blogs, people who have emailed me and reading various forums, my conclusion is that the UW assessment tests is just as good as the NBME assessment tests, so far. Although there have been some observations that UW tend to be overestimate your scores in comparison to NBME, this does not seem to happen in all cases and the score difference is not too big. So all in all, I believe the UW assessment tests have enough track record by this time that we can safely say, they are fairly accurate in predicting USMLE scores. But as in all assessment tests, correlation is never 100%, therefore expect some deviation from predicted scores in the final result.
August 19th, 2009 at 11:04 am
Hey, Nice blog u have here, wish I had found it earlier I just started USMLE World, I’ve done like 6 tests now, 66-72. I got a little bit confused about registering and confirmation of materials by ecfmg. I thought I had up to 2 months to continue with my preparations, but they sent me a mail yesterday that my eligibility period is Aug 1 to Oct31. The first problem is I will be reloating to the UK On the 14th of september from Nigeria. I picked the Africa testing Region. I already sent a mail to ecfmg asking if I can change my region but they will take up to 15 days to reply me. Is it possible to change my testing region for the step 1. Also, apart from the 66-72,% score so far on USMLE World, I had 65% on kaplan Diagnostic and 86% on the free USMLE 150 pratice questions. I’m confused as to what I am confused about. What do you think.
August 20th, 2009 at 7:13 am
Hi olumide,
For the past few years, everyone I know who tried to change testing region were turned down by ECFMG. However, I personally never went through the process, so all I know is from what other people had told me or written in forums. People wind up having to fly out to a different region just to take their exam. But my suggestion is you wait for reply from ECFMG. Who knows, they may have changed policies recently. If they do allow you to change testing region, would appreciate it if you would report back here so others in similar situation in the future can be helped.
Now for the good news. based on your UW scores and Kaplan Diagnostic scores, you seem to be doing very well, especially if you still have more than a month before the exam. Did you do your UW random, mixed, timed or by subject or tutorial mode? Usually, you score lower in UW than in Kaplan. however, both scores indicate somewhere high 80’s to low 90’s. With over a month left to review, you have the potential to get a really high score. Important to go through one more revision of the subjects, but do not study anything new. Do all the q’s preferably timed, random mix blocks and take notes.
Askdoc
September 13th, 2009 at 12:00 pm
Hi, In response to changing testing regions.
I did successfully change my testing region from Europe to North America. In this case, they had to send me a brand new scheduling permit and cancel the old one. However, I did not have to resubmit any documents. It took them 2 weeks to send me my new permit.
September 19th, 2009 at 7:51 am
Hi KluverBucy,
Thanks for the info. Really appreciate it.
Askdoc
September 24th, 2009 at 7:39 am
Hey askdoc,
First I just want to say a huge thank you for putting up all these info about the step 1. And I had a few questions:
For these NBME forms, when you pay yo get them you get all 6 right? Or is it pay per form?
Also, I am scheduled to take the step 1 in May of 2010..so about 8 months from now. And I know I have a knowledge base problem because I would cram for tests and pretty much barely pass through my classes (and believe me I am regretting and hating myself for doing that). But what is done is done and I have realized the grave situation I am in and thus I am wondering is it possible to attain average and mayb even a little bit above average (220 or so) with the amount of time I have between now and May 2010? I am thinking about trying to go through the big robbins text while annotating/note-taking into First Aid along the way and then try to branch off to other topics as it is relevant (immuno/micro, anatomy, biochem, cell bio, etc.)
I was planning to take a NBME form 4 before any studying just to see where I am at right now, and then I was going to take another NBME form 5 about 4 months into studying to see how much progress I’ve made and then I was gonna do NBME form 6 about 2-3 weeks before Step 1 date. Is that a good plan?
Thanks alot for your help askdoc!
October 2nd, 2009 at 3:37 pm
Hi Ming.
First, NBME is paid per form. So it will be expensive to use all 6 forms. For most people 8 months of prep is more than enough to do well. However, how well you do depends on a lot of factors. Your memory capacity, your analytical ability and how well you study will limit the top score you can achieve. Your reading speed, intelligence, ability to concentrate, ability to retain information and how fast you tend to forget information will affect the study period you need to achieve your maximum potential.
Although studying Robbin’s is a very good way to get a high score, what you are planning to do have some problems. First you are relying on your own judgment on how much and what you need to study in order to achieve your desired score. There is no problem with that if you are good at judging how much or how well you need to know the topics to achieve your desired score. However, my experience is that 80% of people do not really know how to do that. That is why I talked about substituted judgment in the What to Study for the USMLE posts. Substituted judgment is when you depend on other people’s judgment to decide what you need to study and how much you need to study to achieve your scores. You do this by using specific resources or enrolling in specific courses that have been tailored to help you achieve a certain performance.
For example, Kaplan Notes contains enough information so that if you retain (not just read or study) >85% of it, you have a good chance of > 220 score. However, even if you retain 100% of FA, I doubt you could score higher than 210. If you just read through the material once or twice, chances are you will just retain anywhere from 20 – 60% of what you’ve read which may result in failing the exam. Retaining around 80% of Robbins (Of course it has to be the correct 80% as Robbin’s include low yield materials and you need to separate the chaff from the grain so to speak) you can get > 230 but only on the pathology portion. Not on the whole USMLE Step 1. So your final score still depends on how you covered the 6 other subject.
For example, my prep course is designed so that if you retain 80% of the material, you’ll probably score > 230 and if you retain 90% of the material, you’ll probably be scoring > 250. It includes lectures which teaches you how to study in order to achieve your goals, plus study materials that emphasize what to study and how much to study to reach particular scores.
So it is very hard for me to assess if what you are planning will achieve the scores you desire. It depends on how good your judgment is.
When you take your first NBME depends on how much reviewing you have finished and not on a fixed time period. For example, my course requires 3 revisions minimum to pass. (some people can pass with just 2 revisions, but a lot won’t. I took 3 revisions to get my 256. Others may need more than 3 to get a 99) My students take their first NBME after the third revision and at least half of qbank if they are using only one online qbank. If they decide to use both Kaplan and UW qbank, the first NBME is after third revision and completing one online qbank.
Askdoc
October 13th, 2009 at 12:13 am
hallo askdoc,
thanks for such great information regarding step1. im gving my step1 in nov 09.how should i plan taking other step exams so that i can apply for 2011 match. i m done with UW(tutor mode) and my scores are in range of 50-60%. any suggestion to increase my scores in step1.
thanks…
November 3rd, 2009 at 5:54 am
Hello Doc, I have been preparing for the USMLE step 1 for about 9 months now, I studied from Kaplan, Goljan and FA, did USMLE World and scored 57%. My exam is in 15 days. I took NBME 2 and scored a 560(236 on the real deal according to the chart). I Plan on taking NBME 3 a week prior to the test. I’m aiming for a >90 grade. My question is if with that score on NBME 2 i have any degree of certainty i can score >90? Another thing is that i did really well on the UW Self assesment with i took 2 months and 1 month ago. I got the same grade (216…95) on both. Thanks in advance doc.
November 3rd, 2009 at 9:20 am
Hi Dr. Brightside,
Just curious, are you using the online NBME assessment test, or one of those downloaded one. If it is the downloaded one you have a big problem. One reason I asked is you have a 57% in UW qbank and that is equivalent to at most a 75. One of my students got a 56% in UW qbank and was told by me that there is a good chance of failing the exam. Took one of those “hard” copy NBME, despite being warned about the lack of reliability unless using the online version , got a 228. So sat for the exam with confidence and failed the actual exam. So before giving you any suggestion, I would like to ask which exam you actually took.
One more thing, if 57% is enough to get a 236, then one of my current students who is now averaging mid-70’s in the qbank would be equivalent to what score? as of now, he is still doing additional revisions to insure he will get a 99. 2 of my students who were averaging mid to high 60’s in the qbank got high 80’s to low 90’s. One student who got average 59% barely passed the exam.
Askdoc
November 3rd, 2009 at 10:22 am
Thanks for the quick reply doc . I used the online form. I had never entered any forums before although i recognized some of the questions for their content and not because i had been asked those questions before. One thing I should mention is that I didnt really memorize the First Aid prior to doing the Usmle World Questions (which i also took online and not the pirated version). 57 was my final average but during the last weeks I was constantly above 60%. Once I was done with World i read First Aid once again and realized that had I memorized most of the things in there I would have done a lot better in World, I didnt really do well with any of the straight recall questions, so that’s what i basically did prior to the NBME i took last sunday.
November 6th, 2009 at 12:46 am
Hi Dr. Brightside,
An average of mid-60’s is more consistent with a low to mid-90’s predicted score in the NBME. Remember, there is no assessment test that is 100% foolproof. Therefore, the more parameters you use to gauge your readiness (NBME and UW in this case) the better the chance that you will get that score. So based on those points, you have a very good chance of hitting mid-90’s. Read up on What to do on the day of the USMLE exam and you are good to go. Good luck on getting your dream score.
Askdoc
November 9th, 2009 at 7:16 am
Thanks Doc. I’ll let you kno how i did once I get my score
November 18th, 2009 at 9:42 pm
hallo askdoc,
i scored 248 in usmle world self assesment form 2 and 230 in form 1. only 2 weeks are left for exam.
November 19th, 2009 at 11:00 am
HI, i’m planning to take my exam in end january , I’m doing a 3rd revision of the subject should complete it in the 1st week of december and I’ll do the kaplan q bank after it then I’ll do the online NBME…. then I’ll do the UW .
so I can say I’ll spend the whole month ( december) in doing q banks( kaplan + UW ) then in january I have 20 days left to do another revision ( 2 weeks ) then in the last 5 days I’ll go over the 1st aid and my notes from the q banks.
I don’t know if this is a good schedule??? I heared doing the questions are the most important …so do you think it is a good idea to spend the last 20 days before the exam doing another revision or I should repeat the UW or there is no pointe to do that , I’m confused. I’m thinking of doing both like spend half of the day going over the UW blocks and reviewing the wrong answers then the other half in going over the books quickly , do you think this is better???
November 19th, 2009 at 11:14 am
please one more question…..I’m planning to take 95 or above I hope.
so what the average scores in kaplan and UW to get such a mark??
and if I’m planning to do one NBME which one to choose?? I”m thinking of 3 or 4. and if my score is not good I’ll do another one but if it is good + scores in kaplan + UW , one is enough.
thank you so much in advance.
November 19th, 2009 at 11:23 am
Hi docjay,
That is good to hear. However, my statistics on UWSA is that they tend to overestimate scores and correlation is a bit wide. However, based on those scores, you will probably do high 90’s with a very small possibility of a 99. Spend the next 10 days doing high yield review. Also if you can do at least 1 NBME assessment, so much the better so you can get an accurate assessment of your current status. Another thing, if you did UW Qbank or Kaplan Q bank, an average of > 72% in the last 10 blocks, timed random mixed will correlate with a high 90’s score, too.
Askdoc.
November 19th, 2009 at 11:30 am
Hi YK,
That is a good schedule. However, I suggest the following. Start with UW qbank rather than Kaplan Qbank. Do 4 blocks per day. Reason is you need to review the questions afterward. So for every block you need to reserve at least 2 hours. One for doing the block. One for reviewing the answers. Plus if you have any really bad weak point, you need to go back to revising that particular topic in your notes or text. Then do an NBME. If the score you get is good enough. Do Kaplan qbank concentrating on simulating exam conditions. make sure to have at least one day where you simulate 7 blocks in 8 hours including controlled breaks. You should spend the last 2 weeks, doing high yield review. Do not redo the qbanks at this stage, since you’ve done your review in December. take a break on the last 2 days before the exam to make sure you are in tiptop shape. Read my post on What to do on the Day of the USMLE exam in this blog.
Askdoc
November 19th, 2009 at 11:38 am
Hi YK,
OK, for a 95 or higher, the minimum average for the last 10 block, mixed, timed and random should be at least a 70. However, if you want to be sure, then go for around 72%. There will probably be a 2 to 3 percentage point difference between UW qbank and Kaplan qbank. So I would say around 70 in UW around 72 in Kaplan.
Askdoc
November 19th, 2009 at 11:45 am
thank you , but I already purchased the kaplan for the next month…
so what the pointe of doing UW before kaplan.
what if I didn’t get a good score in the NBME after the 3rd revision + one of hte q banks??? do u recomend to do the other Q bank then the final reciew before the exam…. or should do a 2 weeks review then the q bank then the exam???
November 19th, 2009 at 12:48 pm
one more thing doc. I’m planning to do 3 blocks in a day cause I need at least 3 to 4 hrs per block cause I’m reading the explanations , I heared they are important , do you think this is a good thing or I’m wasting alot of time and should go over the blocks quickly and spent the other time in reviewing the texts.
November 21st, 2009 at 6:56 am
my score in kaplan simulated exam was 69.should i solve kaplan q bank now..or stick with revision of notes.
November 24th, 2009 at 10:51 am
Hi docjay,
69 in Kaplan should be about mid-90’s. So if you are satisfied with that than there’s no need to do kaplan qbank. But with 2 weeks to go, I believe you’ll benefit more with revision of notes than doing additional qbank. 3 to 4 weeks then qbank is advised.
Anyway, good luck on your exam. Remember to read my post on What to do on the day of the USMLE exam. You will have a good score. Maybe not a 99 (Maybe a 99) but a good score nonetheless.
Askdoc
November 24th, 2009 at 10:53 am
Hi YK,
Read the explanations. In fact even if your answer is correct, read the explanations if it took you a long time to answer that particular question. Also in cases where you are down to last 2, even if you guess correctly, you have to read the answer. For q’s you have no problem with, don’t even bother. For concepts you had problems with in the first 2, be sure to go back and read the text, but only for those concepts you have problem with, OK.
Askdoc
November 28th, 2009 at 3:27 pm
hi ask doc
i did my step 1 and i got my results back, i failed and got 182– soo close but definitely not wnat i wanted.. so in my last month before the exam i memorized FA as well as i could and i always forget things i dont think my memory is that good, but anyway, now im studying with a plan to do usmle world 2-3 times, write in explanations etc i dont know into my FA and do an assessment exam every weekend and then go over that and learn from it, im doing 2 blocks everyday after reading a chapter in FA ( same subject, same questions).. last week my exam mark was 196 this week its 195, i really need to improve this, it’s been 2 weeks since i have been studying again, is there something i can do ? please help, i cannot fail this time:(
i really need to do well to compensate for this failing grade:(
btw i have 49 percent of the u world qbank left to do and the practice exams i did were uworld assesment form 1 and 2
thanks
December 2nd, 2009 at 11:51 am
hi plz helppp,
I don’t know if you will be happy to hear this. But what you are doing will just lead to frustration and in fact is the worst way to study. I know I would fail the USMLE if I used your method. There are 3 things you need to know in order to be able to prep properly. First, is What to Study and your choice of study materials and what you propose to do shows you have no idea what you need to study. Second, you need to know How Much to Study. The fact is you are trusting that combining UW and FA is enough for you to pass shows you do not know How Much to Study. Lastly, you need to know How to Study. The methods you are using clearly show you do not. And judging from your results and methods. The best you can do on your own is barely pass this exam.
Of course, since you have failed your first one, just passing the exam is not an option if you want to match later. You need to get a high score. The question is can you do it on your own. You have 2 options, learn how to prep on your own, or have somebody tutor you on how to prep. For the first option, all the materials you need are in my blog and in the various forums. In fact that is what I did. Learn how to prep properly and then prepped properly. For the 2nd option, you need to either find a very good and knowledgeable friend who is willing to sit beside you for about 4 to 5 months to couch you on How to Study, or you need to enroll in a prep course. But make sure it doesn’t just give you a bunch of reviewers to study but actually teaches you how to study it.
Since you’ve failed your first exam, I doubt you will be able to study on your own. Some people fail the first exam because of overconfidence or unavoidable personal problems. They can usually do it a second time. But if you had prepped the best way you know how the first time, I don’t think it’s possible to do the same thing and get a different result. You have to learn how to prep properly first. I cannot just give you a short prescription and magically it will help you pass. To illustrate, my six one-hour lecture on How to Study in my prep course (Excluding what to study and how much to study) has morphed into 6 1.5 hour lectures already. And from what gets discussed in the group chat session. Those 6 1.5 hour lectures may turn into 10 or even 12 – 1.5 hour lectures soon. Again this just covers How to Study. So I doubt in your case a few words of advise will work.
Askdoc
Just a quote and word of advise.
“You learn more quickly under the guidance of experienced teachers. You waste a lot of time going down blind alleys if you have no one to lead you.”
– W. Somerset Maugham, The Razor’s Edge, 1943
December 4th, 2009 at 5:09 pm
hi
im sorry but i find it confusin to find where you have put the information regarding how to study, can you please direct me to it? i dont have much time left and i really need to study the right way:(
thanks
December 9th, 2009 at 7:35 am
Hi plz helppp,
There is no one article on How to Study that exists. You need to do your own research on How to Study and How to Study for USMLE Step 1. My prep course is the only course or resource I know that covers How to Study for the USMLE Step 1 via 6 1.5 hour lecture or a total of 9 hours. Even then it is evolving and may eventually extend to 18 hours. If I were to write a book on it, it will probably be about 300++ pages long. When I started my prep 4 years ago. I actually went around trying to look for a book that will teach me How to Study for the USMLE Step 1. I was willing to pay hundreds of dollars, if only I can find one that works. People said FA is the one. So I spent ~ $40 to buy it only to find out it doesn’t. So I bought other books. After spending hundreds of dollars on books, I still can’t find one that teaches me How to Study. What to Study, yes, but nothing about How to Study. So I did my own research, starting with exam experiences of people in the various forums. However, there was a lot of disinformation going around, so you need to know what are outright lies and what has a grain of truth. You would be surprised at how many people followed fraudulent advise just because they were what they want to hear. So they wind up failing the exam or scoring low. So after about 6 months, I had a master plan on How to Study the USMLE Step 1, but the only way I will know if it will work or not is after I have finished my prep. So now I know it works. And one year after I started my prep course, I know it works for others, too.
To get you started, go to my forum at http://forums.askdoc-usmle.com and read the posts under Askdoc’s method of USMLE Review. Follow the sequence. That will provide you with a brief overview of where to start. It is by no means complete of detailed. As I said, if I were to write a book, it would be at least 300 pages long. The posts are at most a couple of pages long, but it is a good start. The rest you have to research.
Askdoc
December 29th, 2009 at 4:50 pm
Hi Askdoc,
I’m a foreign grad who just did step 1 exam and have nt recieve score yet.
Well i agree that the irrational idea of “1st Aid +UW ” is nonsence,and i felt so in my exam.
I appreciate the NBME for their new more comprehensive smart USMLE which can really gauge the REAL doctor from PRETENDERS !.
Plus i really misunderstand those who keep thinking of the exam as exam ;they miss the idea of learning for their future to save themselves from liability of error in USA !
accept my regards,
much blessing
March 9th, 2010 at 8:36 pm
hello askdoc,
im preparing for my step 2 ck. i scored 243 on uw sim exam and my score on uworld is 67%.