Abstract
Background: Nearly every state offers loan repayment (LRP) and some offer loan forgiveness to clinicians who commit to work in safety net practices. The effectiveness of these programs from the perspective of safety net practices is largely unknown.
Objectives: To assess safety net practice administrators’ assessments of key outcomes for the 3 principal types of state service programs: LRPs funded by states, LRPs funded jointly by states and National Health Service Corps, and loan forgiveness programs.
Subjects: Administrators of safety net sites where primary care, behavioral health and dental health clinicians began serving in 26 state service programs in 14 states from 2011 to 2018. Survey responses were received from 455 administrators reporting on 754 of 1380 clinicians (54.6%).
Outcome Measures: Administrators’ ratings of their sites’ difficulty recruiting clinicians; relative ease, quickness and cost of recruiting the participating (index) clinician with the service program; program expected effects on participants’ retention; participants’ job performance.
Results: Most administrators (66.1%) reported that recruiting clinicians of the index clinician’s discipline is generally difficult but made easier (81.7%) and quicker (65.4%) with the service program, but only sometimes less expensive (34.8%). 78.8% of administrators anticipate that the clinicians will remain longer because of program participation. Participants are perceived to practice good quality care (96.9%) and be positive contributors (92.4%). Administrators’ assessments are generally similar for the 3 types of programs.
Conclusions: Administrators of safety net practices generally perceive states’ loan repayment and loan forgiveness programs succeed in helping them recruit and retain good clinicians.
- Health Personnel
- Health Policy
- Health Surveys
- Medical Education
- National Health Service Corps
- Practice Management
- Primary Health Care
- Program Evaluation
- Rural Health
- Workforce
Introduction
Nearly every state sponsors programs that help recently trained clinicians pay down their education debt if they contract to work in chronically understaffed safety net practices.1 Many clinicians count on these programs with education debt now substantial for all disciplines, averaging $300,000 for dentists and a median of $200,000 for physicians.2,3 States offer several types of programs that provide debt relief for clinicians while simultaneously providing a recruitment incentive for safety net practices. This study assesses outcomes for the 3 most popular program types from the perspective of the safety net practices these programs are intended to help.
The most common program approach is loan repayment, in which programs repay a portion of clinicians’ education debt for each year they work in an eligible practice. The most common group of state loan repayment programs, currently offered by 42 states, are those cofunded by states and the National Health Service Corps (NHSC) as part of the NHSC’s State Loan Repayment Program.4,5 Unlike the NHSC’s much larger, federally administered Loan Repayment Program,6,7 state-NHSC LRPs are administered by participating states but operate under a common set of federal rules, including a 2-year minimum initial service commitment for participants, a substantial penalty for those who fail to complete this commitment, and service restricted to outpatient settings in Health Professional Shortage Areas (HPSA).5,7,8 States otherwise can individualize their programs.
A second group of loan repayment programs is created, funded and administered entirely by states. Funding is typically from state revenues but sometimes matched by the communities and practices where clinicians serve.9,10 With these solely-state LRPs, state legislators and program leaders are completely free to set program rules to meet their state’s particular needs.10
Fewer states sponsor a third type of program, loan forgiveness. These programs recruit individuals earlier when they are still students by offering education loans that come with an upfront guarantee that a portion of the loan amount will be forgiven if after training they work in qualifying clinical sites.10
Forty-nine states now offer 1 and often several state-NHSC LRP, solely-state LRP and/or state loan forgiveness programs, collectively called here state service programs.1,5,9,11,12 Despite their popularity with states, state service programs have seldom been evaluated and the relative successes for the 3 types are unknown. To our knowledge only 1 study, published nearly 20 years ago and solely of physician participants, has assessed outcomes across states for the various program types.13 No study to date has assessed outcomes of state-NHSC LRP programs as a unique group.
Among the unknowns about states’ service programs is how they are viewed by the safety net practices where clinicians serve. Practices typically embrace these programs and tout the availability of loan repayment to job applicants.14 Practice administrators help clinicians qualify for programs by providing the attestations required by the state agencies that manage these programs. Practice administrators are well positioned to report on their sites’ experiences with these programs, indeed many work daily with participating clinicians and come to know them as employees, professionals and people. Administrators generally approach these programs from the viewpoint of the needs of their organizations and communities, so they can provide a “customer perspective” on states’ service programs.
In this study we use survey data from administrators of safety net practices in 14 states where clinicians of a variety of medical, dental and behavioral health disciplines participate in 26 state service programs. We assess how the state-NHSC LRPs, solely-state LRPs, and state forgivable loan programs in these states differ in administrators’ perspectives on key program outcomes, specifically their contributions to sites’ recruitment efforts, retention efforts, the job performance of participating clinicians, and the quality of sites’ interactions with programs.
Methods
Subjects
This study uses questionnaire data from the Provider Retention & Information System Management (PRISM), a collaborative of states that routinely collects and reports feedback from clinicians and practice administrators involved in state and federal education debt support-for-service programs.15,16 The Collaborative’s purpose is to help states better understand and strengthen their clinician workforce programs through data. States’ lead agencies in the Collaborative are generally their Primary Care Offices 17 or sometimes other agencies central to states’ clinician distribution efforts. This study uses only administrators’ survey response data and only for states’ programs.
Data
As part of the Collaborative’s ongoing work, questionnaire invitations are emailed to administrators each year on the anniversary of the start date of each clinician’s service program contract. The administrator surveyed is typically the chief executive officer or clinic manager, or sometimes a human resources director.
This study uses administrator responses from sites where clinicians were serving program contracts with start dates from January 1, 2011 to September 1, 2018 in all 26 state service programs in 14 of the Collaborative’s 22 states as of 2018. Eight states of the Collaborative were not part of this study, 6 because they only include federal programs in their survey work and 2 because they only survey participating clinicians and not administrators. Of the 1380 service contracts with these 26 programs commencing during this 7-year period, by October 2019 (before the onset of the COVID-19 pandemic) 455 administrators had completed 1 or more annual questionnaires for 754 clinician-contracts (54.6% overall participation rate) for clinicians working at 553 sites. Questionnaires were completed for 413 solely-state LRP contracts (54.8% response), 281 state-NHSC LRP contracts (53.1% response) and 60 state forgivable loan program contracts (61.9% response).
In the online questionnaires, site administrators are instructed to respond with respect to a named clinician-participant (the “index clinician”) participating in a named state service program at a named principal service site. Site administrators use 5-point response scales with a neutral central response option to (1) characterize how easy or difficult it has generally been for the site to recruit clinicians of the index clinician’s discipline, (2) estimate how much easier or more difficult, quicker or longer, and less or more expensive it was to fill the index clinician’s position because of the availability of the state service program, and (3) indicate if they believe that participating in the state service program will prompt the index clinician and also other clinicians of that discipline to remain longer or shorter in the practice than if not participating. Administrators also rate how well the index clinician is fulfilling key aspects of their job, and rate their practice’s involvement with the state service program. Confidentiality is promised for assessments of clinician performance and sites’ program involvement.
Analysis
In analyses, we used data from only the most recently completed administrator questionnaire for each clinician within a given program and contract. In cases where the most recently completed questionnaire was missing specific data items, we pulled in data from annual questionnaires completed previously by that same administrator for that clinician, program and site, when available. Less than 2% of data remained missing for administrators’ assessments of the service programs’ recruitment and retention effects, and less than 6% were missing for assessments of participating clinicians’ performance and ratings of sites’ interactions with service programs. Missing values were not imputed.
Analyses characterize the disciplines of participating clinicians, the type of sites where participants serve, and aspects of the health insurance coverage and race-ethnicity of patients of these sites, per administrators’ reports. Differences across program types are tested with Pearson χ2 tests or analyses of variance, as appropriate. Scaled item responses are dichotomized 2 ways to present both (1) percentages of administrators reporting the 2 positive responses combined (eg, “somewhat easier” and “much easier”) and (2) percentages reporting the 2 negative responses (eg, “somewhat more difficult” and “much more difficult”), with differences across the 3 program types compared with Pearson χ2 tests. All percentage figures are weighted for differences in response rates found across states and the 3 types of programs. Weights varied from 0.70 to 1.74; the calculated design effect due to weights is 1.065. Response rates did not differ across disciplines and types of practice site (P = .63 and 0.11, respectively) and are thus not included in weighting.
Logistic regression models then retest program outcomes found to differ for the 3 service program types in bivariate analyses. Models adjust for the differences across the 3 program types in their participants’ disciplines, types of practices and characteristics of practices’ patients.
All analyses are performed with IBM SPSS version 26 (IBM Corporation; Armonk, NY). Because some administrators reported on their practices’ experiences with more than 1 clinician, we applied the Complex Samples feature within SPSS to all bivariate and multivariate analyses to account for this clustering within the data. The p-value of statistical significance was set at .05. This study using previously collected and deidentified data were determined to be exempt from human subjects review by the University of ??? Office of Human Research Ethics (Study 19–2397).
Results
Description of State Programs, Participants, and Service Sites
Across the 14 states, site administrators completed questionnaires reporting on their experiences with 413 clinicians participating in 10 solely-state LRPs, 281 clinicians in 13 state-NHSC LRPs, and 60 clinicians in 3 state forgivable loan programs (Table 1). Administrators were reporting most often on their sites’ experiences with physicians (n = 208; 27%) and least often on clinicians of the behavioral health disciplines (n = 88; 13%) and “other disciplines” (n = 88; 13%) (Table 2). Administrators where state-NHSC program participants worked reported on proportionately fewer physicians and more behavioral health disciplines than administrators reporting on participants of the other 2 types of programs (P < .001).
List of State Service Programs by Type of Program With Number of Completed Administrator Questionnaires in Analyses
Disciplines, Service Sites and Special Population Groups, by State Service Program
For the 3 program types combined, administrators most often reported on the experiences of Federally Qualified Health Centers (FQHC) 18 and other community health centers (CHC) (n = 228; 29%) (Table 2). Administrators with state-NHSC LRP participants were more often in FQHCs/CHCs and less often in “other office-based practices” than administrators reporting on participants of the other 2 types of programs (P < .001).
Among all practices combined, 52.8% of patients were reportedly insured through Medicaid, Indian Health Service, tribal insurance, or were uninsured, somewhat lower for sites where clinicians served in forgivable loan programs (P < .001). For all practices combined, a reported 32.5% of patients were Black, Hispanic and Native American or Alaskan Native, somewhat higher among sites with solely-state LRP participants (P < .001).
Perceived Contributions of State Service Programs to Sites’ Recruitment and Retention of Clinicians
Among all practices combined, 2/3 of administrators (66.1%) reported that recruiting a clinician of the index clinician’s discipline is “fairly difficulty” or “very difficult” without the help of the service program, and 14.5% described recruitment as “fairly easy” or “very easy” (Table 3). The remaining 19.4% reported that recruiting this discipline is neither easy nor difficult (data not shown in the table). Administrators where state-NHSC LRP clinicians served were most likely to report that recruitment of their participating clinicians’ discipline without the program was difficult (P = .019).
Weighted Percentages* of Administrators’ Perceptions of the Impact of the State Service Program on Their Site’s Ability to Recruit and Retain Clinicians of the Participating Clinician’s Discipline, by the Three Program Types
Overall, 81.7% of administrators reported that the recruitment of the index clinician was made easier because of the state service program incentive, 2/3 (65.4%) reported that recruitment was quicker, but only 1/3 (34.8%) reported that recruitment was less expensive, with no statistical differences across the 3 program types. A significant minority (9.6%) of administrators reported that the state service program made recruitment of the index clinician more expensive, most frequently among site administrators that had recruited participants of state-NHSC LRP programs (P = .002).
Three-quarters of administrators (78.8%) believed that the state service program will help their currently participating clinician remain longer in the practice than if they were not participating, and 83.0% reported that program participants of the index clinician’s discipline generally remain longer than nonparticipants, with these percentages somewhat lower for participants of forgivable loan programs (Table 3). Very few administrators anticipated that state service program participation makes site retention shorter for either the index clinician or others of their discipline.
Perceptions of Participating Clinicians and Sites’ Involvement With Service Programs
Site administrators’ perceptions of how well program participants fulfill their job requirements were quite positive (Table 4). A strong majority of administrators responded “well” or “very well” on how their participating clinician was meeting expectations for providing good quality care (96.9%), being a positive contributor within the practice (92.4%), and for their overall job performance (93.8%), with no significant differences across the 3 types of programs.
Weighted Percentages of Administrators’ Perceptions of the Job Performance of Participating Clinicians and the Site’s Involvement with Service Programs, by the Three Service Program Types
Two-thirds of administrators rated their practice’s involvement with the service program as “very good” or “excellent” when clinicians were being hired and qualifying for the service program as well as when they were participating in the program. Similarly, two-thirds rated their site’s overall participation with the program as “very good” or “excellent.” Again, ratings were similar for the 3 program types.
A minority of administrators responded “fair” or “poor” for their practices’ involvement with the service program during the clinician hiring and program qualifying phase (7.6%), as clinicians participated in the program (5.2%), and for their site’s overall participation with the program (6.0%). Administrators where forgivable loan program participants worked more often rated interactions with programs when clinicians served as fair or poor (P = .036).
Adjusting Outcome Differences in the 3 Types of Programs for Differences in Their Participants and Sites
The 4 differences in outcomes across the 3 service program types found in bivariate analyses remained after using logistic regression models to account for differences in their participants’ disciplines, types of practices and patient race-ethnicity distributions (Table 5). Logistic regression of differences across program types in likelihood of poor interactions with practices when clinicians serve generated a statistically nonsignificant model (P = .08), which is therefore not included in Table 5. Specifically, compared with administrators of practices where solely-state LRP participants worked, logistic regression models still found that practice administrators where state-NHSC LRP participants worked most often reported baseline difficulty recruiting clinicians of their participant’s discipline (odds ratio 1.64, P ≤ . 05) but also that the program made recruitment more expensive (odds ratio 3.25, P ≤ .001); and practice administrators where state forgivable loan program participants worked were less likely than administrators where solely-state LRP participants worked to believe that programs will help their current participants (odds ratio 0.52, P ≤. 05) and other clinicians of their discipline remain longer in the practice (odds ratio 0.44, P ≤ .05).
Adjusting Outcome Differences† Found across the Three Types of State Service Programs in Bivariate Analyses for Differences in Their Participants’ Disciplines, Types of Practice Sites, and Patient Race-Ethnicity
Discussion
Loan repayment and forgivable loan programs are popular with states to help recruit and retain clinicians in safety net practices. This study assesses if these programs succeed from the perspective of these safety net practices, providing a “customer perspective” on these programs.
According to this study’s 455 practice administrators reporting their practices’ experiences with clinicians participating in 26 programs in 14 states, these programs generally succeed in their intended outcomes. Most administrators report that recruiting clinicians of their participants’ disciplines is difficult for their practices without these programs, and most report that programs made recruiting easier and quicker but only sometimes less expensive. Most administrators anticipate that program participants will remain longer in their practices because of their participation, and nearly all participants are seen to be good clinicians and positive contributors. On the other hand, about 14% of administrators indicate that recruiting their participating clinicians’ discipline is easy even without the service program, 10% report that the service program makes recruitment more expensive, and 6% report that their practices’ interactions with the service program has not been positive overall.
These administrators’ generally positive experiences with states’ loan repayment and loan forgiveness programs mirror the generally positive experiences of clinician participants. Prior studies find that more than 9 out of 10 clinicians complete their state service program contracts and report they would again enroll in their programs if they had it to do over.13 A great majority of participating clinicians are also satisfied with most aspects of their work and practices while participating; indeed, they are typically more satisfied than nonparticipating clinicians in comparable practices.13,19
Administrators’ general expectation of longer retention of participants of all 3 types of state service programs than nonparticipating clinicians is consistent with the long retention anticipated by the clinician-participants themselves of many of these same programs.19 Similarly, in the late 1990s primary care physicians who had participated in 29 state service programs demonstrated longer retention in their practices than a matched comparison group of nonparticipating physicians.13 This earlier study also found that average retention for participants of solely-state loan repayment programs was longer than for participants of loan forgiveness programs, a difference mirrored in the experiences of the current study’s administrators.
States can choose to offer loan repayment programs operated jointly with the NHSC, loan repayment programs they operate themselves, and/or forgivable loan programs. This study finds that within these 14 states these 3 groups of programs are generally similar in the types of clinicians they support and types of practices where they serve. The principal differences are that state-NHSC LRPs support proportionately fewer physicians and more behavioral health providers, which reflects the NHSC’s emphasis on fielding a diverse workforce,20⇓–22 and their participants more often serve in FQHCs, which meet the NHSC’s requirement for its clinicians to serve in HPSAs.21
Practice administrators also generally perceive these 3 types of programs to be similar in their outcomes. Specially, perceptions are similar of the recruitment and retention benefits they provide to safety net sites, clinicians’ job performance, and the quality of interactions between programs and practices. Relatively small differences were found with administrators of state-NHSC LRP sites more often reporting recruitment being difficult without the program but also that recruiting is more expensive with the program. Fewer administrators of forgivable loan program sites believe that program participants remain longer in their service practices because of the program.
We wondered if some practices find it more expensive to recruit clinicians with state service program incentives when programs require practices with participating clinicians to cover a portion of the cost of the financial benefits clinicians receive.10 This is the case for 5 of this study’s 10 solely-state LRPs, 8 of the 13 state-NHSC LRPs, but none of the 3 state forgivable loan programs. In post hoc analyses we find that administrators who reported on clinicians participating in programs that require dollar contributions from practices or local communities were much more likely to report that recruitment with the program was more expensive than were administrators working with clinicians in programs not requiring local cost sharing (17.5% vs 0.8%; P ≤ . 001). Thus, it is evidently the cost sharing required by some programs that can make recruiting through them more expensive for practices. The recent American Rescue Plan Act of 202123 for the first time eliminated required matching dollars from states for state-NHSC LRP programs,24,25 which should in turn eliminate the costs that some states have passed along to practices and will make recruitment through these programs no longer more expensive.
Even when hiring clinicians participating in state service programs that do not require matching dollars from practices, more than half of administrators report that the recruitment was neither more nor less expensive for their practice when clinicians participate in the service program even though the recruitment was easier and quicker. Perhaps this is because the cost of search firms—practices’ greatest expense when filling clinician vacancies—are the same whether or not the clinician that is hired participates in a loan repayment program.26
Limitations
We believe but cannot know that this study’s findings are fully applicable to state service programs in states that do not participate in the PRISM Collaborative. States with service programs they perceive to be successful could be either more or less likely to participate in the Collaborative. Further, administrators’ experiences with these programs may have changed with the COVID-19 pandemic, which has disrupted these practices and the work and lives of these clinicians.27 Further, this study assessed outcomes for program participants of all disciplines as a group. Administrators’ perceptions of outcomes may differ for various disciplines.
This study draws on information from annual surveys developed for feedback on specific outcomes for these unique programs. Without previously validated items to draw on, items in these questionnaires were developed de novo based principally on face validity and initially tested for clarity and since demonstrated through 10 years of successful use.
States’ forgivable loan programs are particularly difficult to evaluate because there are so few, only 11 nationwide in the mid-1990s.10 An online search now still finds 11 programs, of which this study includes 3. To test if 1 of these 3 might be an outlier affecting findings for the group, we compared all outcomes across the 3 programs. We found that their outcomes differ statistically only in the proportion of administrators anticipating that their current clinician will remain longer in the practice because of program participation.
Conclusions
In the experiences of practice administrators, states’ programs to support the clinician workforce in safety net practices through education debt assistance meet their key program goals. Site administrators’ reports indicate that these programs generally target practices where recruitment is difficult, make recruitment easier and faster, lead to longer retention, support clinicians who are good contributors, and the programs themselves are generally good to work with. Success is overall comparable for solely-state loan repayment, state-NHSC loan repayment, and state loan forgiveness programs.
This study also identifies areas where some state service programs could be strengthened. With one-third of administrators reporting that it is not difficult to recruit the discipline of their current program participant, some programs could better target sites and disciplines for which recruitment is a genuine challenge. And with 1/3 of administrators rating their interactions with programs less than “very good,” programs might find ways to make participation more uniformly positive for practices.
Notes
This article was externally peer reviewed.
Funding: This study was conducted as part of the ongoing work of the Provider Retention & Information System Management (PRISM) Collaborative, which is funded by the Federal Office of Rural Health Policy as well as membership dues from participating states.
Conflicts of interest: Robert Sewell, Thomas Rauner, Marc Overbeck, and John Resendes work in state public agencies that operate or are affiliated with state service programs that are included in this study. Donald E. Pathman and Jackie Fannell have no conflicts of interest to report.
To see this article online, please go to: http://jabfm.org/content/35/5/1015.full.
- Received for publication March 4, 2022.
- Revision received June 3, 2022.
- Accepted for publication June 7, 2022.
In this issue
Jump to section
Related Articles
Cited By...
- No citing articles found.