American Board of Family Medicine In-Training Examination

The American Board of Family Medicine (ABFM) administered the first In-Training Examination (ITE) in 1979 and since then has continued to offer the examination to Accreditation Council for Graduate Medical Education-accredited family medicine residency programs on the first Friday in November. The purpose of the examination is to provide an assessment of each resident's progress while providing comparative data about the program as a whole.

The ITE was offered solely in the traditional pencil and paper format until the 2006 administration, when the ABFM asked residency programs to voluntarily participate in an Internet-based testing ITE pilot project. The pilot was planned as the initial exploration into the possibility of eventually delivering the ITE exclusively in an online format. The initial pilot examination involved 41 Accreditation Council for Graduate Medical Education-accredited residency programs and 633 resident examinees. Two international groups, from Australia and Macau, also participated with a combined total of 12 examinees. The pilot examination was a success; minimal problems were reported, including temporary system lockups and delays in progressing from item to item.

The ABFM continued to offer the Internet-based examination option for both the 2007 and 2008 ITE administrations. In 2007, the ABFM chose to outsource the examination by using a national testing vendor instead of hosting the examination on the ABFM servers, which opened the online version to an even greater number of residents. Unfortunately, significant technical problems were experienced with this examination despite the fact that a trial examination had been undertaken. The official examination day progressed very well until the programs on the West Coast began to log on around 11:00 am EST. At that point, all programs began to experience latency in moving from item to item and many of the West Coast programs were unable to log on at all. Many programs had no choice but to opt out of the online examination and administer the remainder of the items by paper and pencil.

A review of the configuration of the vendor's Web servers, application, database, and error logs was completed by the Microsoft Server Support Team, and it was discovered that the vendor was not at fault. Further investigation indicated that the majority of the problems could be traced to Internet Explorer 6. Microsoft determined that the delay resulted from a hard-coded setting in Internet Explorer 6 that was invoked after a network delay of more than 90 seconds. The problem could not be replicated when using Internet Explorer 7; the longest delays were 5 seconds. Microsoft and the testing vendor suggested that future online participants would be required to use Internet Explorer 7 as their Web browser to avoid the same issues experienced during the 2007 examination.

The testing vendor has since made many upgrades to their testing system, including installation of a new load balancer solution for Web traffic and Internet-based testing database server upgrades. The system was load tested in February 2008 and it was determined that it can support 10,000 concurrent users.

The 2008 administration of the online ITE was largely considered a great success with approximately 1100 residents testing successfully. Minimal problems were experienced, mostly caused by pop-up blockers not being turned off on the workstations and some programs still using Internet Explorer 6, despite the ABFM's advice to use Internet Explorer 7 only.

The ABFM's ultimate goal is to transition entirely to online delivery of the ITE by 2014. Rest assured that we will continue to work closely with the testing vendor to strive toward administering a trouble-free examination. We believe that the online test delivery method is in the best interest of your residents because it will allow them to experience the same computer-based format they will encounter when sitting for their first Certification Examination on completion of residency.