Objective: To investigate whether different risk-adjustment methodologies and economic profiling or "practice efficiency" metrics produce differences in practice efficiency rankings for a set of primary care physicians (PCPs).
Data source: Twelve months of claims records (inpatient, outpatient, professional, and pharmacy) for an independent practice association HMO.
Study design: Patient risk scores obtained with six profiling risk-adjustment methodologies were used in conjunction with claims cost tabulations to measure practice efficiency of all primary care physicians who managed 25 or more members of an HMO.
Data collection: For each of the risk-adjustment methodologies, two measures of "efficiency" were constructed: the standardized cost difference between total observed (standardized actual) and total expected costs for patients managed by each PCP, and the ratio of the PCP's total observed to total expected costs (O/E ratio). Primary care physicians were ranked from most to least efficient according to each risk-adjusted measure, and level of agreement among measures was tested using weighted kappa. Separate rankings were constructed for pediatricians and for other primary care physicians.
Findings: Moderate to high levels of agreement were observed among the six risk-adjusted measures of practice efficiency. Agreement was greater among pediatrician rankings than among adult primary care physician rankings, and, with the standardized difference measure, greater for identifying the least efficient than the most efficient physicians. The O/E ratio was shown to be a biased measure of physician practice efficiency, disproportionately targeting smaller sized panels as outliers.
Conclusions: Although we observed moderate consistency among different risk-adjusted PCP rankings, consistency of measures does not prove that practice efficiency rankings are valid, and health plans should be careful in how they use practice efficiency information. Indicators of practice efficiency should be based on the standardized cost difference, which controls for number of patients in a panel, instead of O/E ratio, which does not.