Do Medical Scribes Help Primary Care Providers Respond More Quickly to Out-of-Visit Tasks?
==========================================================================================

* Leah Zallman
* Wayne Altman
* Lendy Chu
* Sharon Touw
* Karissa Rajagopal
* Steven Dolat
* Assaad Sayah

## Abstract

*Purpose:* Medical scribes are charged with decreasing documentation burden associated with patient visits. Reducing time spent on documentation may afford providers the opportunity to respond to out-of-visit inbox tasks faster.

*Methods:* We compare changes in the time taken to address patient portal messages, prescription requests, and test results from before to after scribe implementation among scribed primary care providers (PCPs), compared with nonscribed PCPs during the same time period. We used generalized estimating equations with robust standard errors to account for repeated measures and the hierarchical nature of the data, and adjusted for provider and patient characteristics.

*Results:* We examined 472,411 tasks, including 27,645 tasks for 5 scribed PCPs and 444,766 tasks of 74 nonscribed PCPs. In unadjusted analyses, we found no change in time to completion for prescription refill requests, results and patient portal messages; the change in time to completion from pre to post intervention among scribed PCPs was 1.02 times that of nonscribed providers (*P* = .585) for prescription refill requests, 1.06 times that of nonscribed providers (*P* = .516) for patient portal messages, and 1.02 times that of nonscribed providers (*P* = .787) for results. Adjustment for provider and patient characteristics did not change these findings.

*Conclusions:* Our study suggests that scribes are not associated with improved time to completion of inbox messages for PCPs. While scribes seem to have many benefits, our study suggests they may not improve time to completion of out-of-visit tasks. Reducing the time to completion for these tasks likely requires other interventions targeted to achieve those outcomes.

*   Community Medicine
*   Documentation
*   Electronic Health Records
*   Patient Portals
*   Primary Health Care

## Introduction

As electronic medical records (EMRs) and online patient portals are becoming more widespread, primary care providers (PCPs) have become responsible for more out-of-visits tasks.1,2 In addition to face-to-face office visits, PCPs must review and respond to medication refill requests, phone and patient messages, and laboratory, imaging, and consultation reports.3 In fact, a quarter of a primary care physician’s time in the EMR is spent on inbox management.4 This electronic inbox management has substantially increased provider workload and created a new source of work-related stress for PCPs.5,6 In fact, for every hour PCPs provide direct clinical face time to patients, they spend nearly 2 additional hours on the EMR and desk work during the clinic day.7

This time spent outside of the visit has significant consequences for both PCPs and patients. PCPs spend 1 to 2 hours of personal time at night doing mostly EMR work,7 such as responding to prescription requests and patient messages. Burnout and stress among providers are increasing,8 leading to high turnover rates.9 In addition, this increased demand on PCP time may also affect patient care. Providers report that patient care may not be optimized because of time constraints and numerous patient care activities requiring response outside of an office visit.10,11

Medical scribes are increasingly used in clinical practice in the United States to address visit-related burdens. Scribes are trained personnel who accompany providers during visits to provide documentation support and to assist with other administrative tasks. Not surprisingly, scribes are associated with improved visit outcomes such as increased productivity,12⇓⇓–15 decreased charting time for providers,12,16⇓–18 and increased face-to-face time with patients.12,15,18

Scribe support during visits may also allow PCPs to more quickly address out-of-visit tasks, such as authorizing prescription requests, managing patient results, and responding to patient messages. That is, by reducing the time providers spend on charting,17 scribes may increase the time available to address out-of-visit tasks. In turn, this may reduce delays in access to care, improve timeliness of care, and improve patient satisfaction and engagement.19⇓⇓–22

While previous studies have focused on the impact of scribes on visit-based processes, no studies have examined their impact on timeliness of responses to out-of-visit tasks. In this study, we examine whether medical scribes were associated with timeliness of responses to out-of-visit tasks. Specifically, we examined the impact of scribes on time to completion for prescription requests, addressing results, and replying to patient messages on the electronic portal.

## Methods

### Design

This observational study compared the change in outcomes before and after scribe implementation between scribed providers and nonscribed providers. We compare visits during the year before scribe implementation (July 2017 to June 2018) with the year after implementation (July 2018 to June 2019).

### Setting

This study was conducted at a safety net community academic health system, which uses an EMR (EPIC, Verona, WI). The EMR integrates inpatient, emergency, and outpatient care, including primary care, specialty, laboratory, pharmacy, and imaging data. In addition, the electronic inbox receives messages that patients send via a secure portal Web site, MyChart. PCPs can access their inboxes on both computers and mobile devices (via EPIC Haiku). Once a provider has addressed a task, the task is either automatically removed from their inbox or it is removed when the provider indicates they have completed the task by clicking a “done” button.

### Outcomes

We defined “time to completion” as the time from when a task arrives in the provider’s inbox to the time a task is cleared from a provider’s inbox. We examined time to completion for prescription requests, results, and electronic portal patient messages. We selected these tasks because responses to these tasks are generally dependent on the provider. In contrast, other inbox messages such as telephone calls and staff messages often result in back and forth with other staff; thus, time to completion is dependent on other staff and would not be expected to be impacted by scribes.

### Intervention

Scribes were recent college graduates who were interested in careers as health care professionals. Scribes received 3 days of training on how to function effectively in their role and 1 day of training in EMR functionality. Scribes continued learning through feedback from supervising PCPs. Scribes documented patient encounters, recording medical and social histories and physical examination findings, and transcribing discussions of treatment plans and PCPs’ instructions to patients. PCP participation was voluntary. PCPs received no compensation for participating in the scribe program, and PCPs were not required to increase clinical care (ie, add patients to their panels or see more patients) to work with a scribe. PCPs that worked with scribes were not statistically different from those who declined scribes, in gender, race, provider type (physician vs nurse practitioner), tenure at the institution and proportion of time working clinically (Appendix Table 1).

View this table:
[Table 1.](http://www.jabfm.org/content/34/1/70/T1)

Table 1. 
Patient Demographics of Tasks Among Visits to Scribed and Nonscribed Providers Before and After Scribe Implementation

### Inclusion/Exclusion Criteria

Because the first year at an institution is a learning period for PCPs during which inbox management skills are increased, we excluded 12 PCPs who worked at the institution for less than 1 year before the scribe program. We excluded 1 provider who was in the postintervention period for less than a month. To focus on providers who received the intervention most of the time, we excluded an additional 12 scribed providers who had less than 66% of their visits scribed during the post period. This resulted in our final sample of 5 scribed providers and 74 nonscribed providers.

We started with 528,742 tasks. Based on 1 PCP’s (WA) extensive clinical experience with scribes, we excluded tasks that occurred in the first 6 weeks of working with a scribe to account for an initial learning period (n = 2943; 0.6%). We removed extreme outliers through a 2-step process. First, we examined the median difference in time to completion by task type pre to post for each provider among scribed and nonscribed providers. Extreme changes in time to task response could indicate random variation; we therefore removed tasks for providers if the difference in time to completion for that task was an extreme outlier (defined as 3 × IQR) (n = 44,501; 8.4%).23 Finally, because providers can keep uncompleted tasks in their inboxes intentionally (eg, as a reminder to follow up at a future date), we removed those with values greater than the 99th percentile (n = 4767; 0.9%).

### Analysis

We compared demographic characteristics for patients and providers using fisher’s exact or the c2 test for categorical variables and *t*-test for continuous variables. Because the outcome of time-to-completion was right skewed, we used a natural log transformation. We first conducted bivariate analyses, comparing outcomes pre to post separately for scribed and nonscribed providers using *t*-tests. Next, we compared the change in outcomes from before scribe implementation to after scribe implementation among scribed providers, compared with nonscribed providers using generalized estimating equations with robust standard errors to account for repeated measures (ie, multiple visits by the same patients) and the hierarchical nature of the data (patients nested within providers). Finally, we controlled for provider characteristics (gender, race, panel size, percent full-time equivalent in outpatient clinical care), patient demographics (age, gender, race/ethnicity), and patient complexity (complex care management program enrollment and at least 1 inpatient admission). We were unable to control for provider type (MD vs NP) due to the small number of NPs in the sample. Although we were able to include many covariates in the model, sample size limitations did not allow us to control for all covariates; thus, we were unable to include provider tenure in the models.

The Cambridge Health Alliance Institutional Review Board considered this study exempt from review.

## Results

### Task Characteristics

We examined 472,411 tasks, including 27,645 tasks for 5 scribed providers and 444,766 tasks of 74 nonscribed providers (Table 1). Refill requests included 178,160 tasks, 74,865 were patient portal messages, and 219,386 were processing results. These tasks were completed for patients who were more likely to be female, older than 21 years of age, have English as their language of care, and be nonwhite.

### Time to completion (Bivariate Analyses)

Time to completion for results did not change among scribed providers (5.43 to 5.67 hours; *P* = .39) or among nonscribed providers (7.56 to 7.67 hours; *P* = .17) (Table 2). Similarly, time to completion for prescription requests did not change among scribed providers (2.20 to 2.24 hours; *P* = .62) or among nonscribed providers (3.04 to 3.06 hours; *P* = .58). Time to completion for patient messages did not change among scribed providers (5.86 to 6.97 hours; *P* = .60) and decreased among nonscribed providers (4.60 to 4.44 hours; *P* = .03).

View this table:
[Table 2.](http://www.jabfm.org/content/34/1/70/T2)

Table 2. 
Time to Task Response (in Hours) Among Scribed and Nonscribed Providers Before and After Scribe Implementation*

### Unadjusted Change in Time to Completion

In unadjusted analyses, the change in time to completion for prescription requests from pre to post intervention among scribed providers was no different among scribed providers compared with nonscribed providers (change among scribed providers was 1.02 times that of nonscribed providers, *P* = .59) (Table 3). Similarly, the change in time to completion for patient messages from pre to post intervention among scribed providers compared with nonscribed providers was no different (change among scribed providers was 1.06 times that of nonscribed providers, *P* = .52). Finally, the change in time to completion for results from pre to post intervention among scribed providers compared with nonscribed providers was no different (change among scribed providers was 1.02 times that of nonscribed providers; *P* = .79).

View this table:
[Table 3.](http://www.jabfm.org/content/34/1/70/T3)

Table 3. 
Unadjusted and Adjusted Models Comparing Change in Time to completion for Out-of-Visit Tasks Among Scribed and Nonscribed Providers Before and After Scribe Implementation

### Adjusted Change in Time to Completion

After adjustment for provider and patient characteristics, the change in time to completion for prescription requests from post to pre intervention among scribed providers compared with nonscribed providers was no different (change among scribed providers was 1.06 times that of nonscribed providers; *P* = .19) (Table 3). Similarly, the change in time to completion for patient messages from post to pre intervention among scribed providers compared with nonscribed providers was no different (change among scribed providers was 1.09 times that of nonscribed providers; *P* = .34). Finally, the change in time to completion for results from post to pre intervention among scribed providers compared with nonscribed providers was no different (change among scribed providers was 1.02 times that of nonscribed providers, *P* = .72).

## Discussion

In the first examination of the impact of scribes on out-of-visit task response, we find that scribe implementation is not associated with changes in time to completion for prescription requests, patient message, and results in primary care. These findings remained true even after adjustment for provider and patient characteristics.

Prior examinations suggest that scribes decrease both the time spent in visits and the time spent charting outside of visits.12,14 Our findings suggest that this time saved due to scribes does not result in quicker responses to out-of-visit tasks. Several explanations for this may exist. Providers may use the time saved during in-visit tasks to address other aspects of clinical care, such as speaking with and calling patients, or consulting with other providers or clinical staff. Alternatively, providers may reduce their time working, resulting in better work-life balance; this theory is supported by evidence that providers are more satisfied and burnout is improved after scribe implementation.12,24 In addition, providers may have spent time reviewing the quality of scribe documentation, and may therefore not have had additional time available to address tasks. Providers may also address inbox items, particularly those that require significant cognitive burden (such as patient messages) at particular times of the day (eg, after patient care), which would limit the impact of scribes on the time to address these tasks. Scribes in this study did not have clinical training; models in which scribes have clinical training (such as medical assistant or nursing) may have more widely felt impacts. Finally, providers may be devoting more cognitive attention to these tasks, which could require more time but also result in better care. Replication of these findings and examination of reasons why scribes may not lead to improved time to completion is a subject for further study.

Our study should be interpreted in the context of several limitations. It was conducted at 1 institution and findings may not be generalizable beyond this setting. For example, in other settings, scribes may have a different scope of work and may help manage inbox tasks. Similarly, the study largely included MDs, who have a different scope of practice than other providers, and the results may not be generalizable to other provider types. It reflects the experience of a small number of scribed providers with a large number of inbox tasks; replication with larger cohorts of providers would help reflect broader experience. Providers volunteered for the program; thus, selection bias may have contributed to unmeasured confounders (such as less efficiency or having other responsibilities). Given the nature of the intervention, providers were not “blinded” to the fact that they were or were not using scribes; it is unknown how this impacted their behaviors. However, since there was no articulated expectation that time to task completion would change, we would expect the impact of this lack of blinding to be minimal. In addition, scribes were not working with PCPs at all visits; the impact of scribes on providers with full time scribe support remains to be seen. In this setting, PCPs manage inbox requests. Thus, our findings may not be applicable in settings where other team members manage inbox requests. Finally, the degree to which our findings are generalizable to other inbox tasks, such as telephone calls and referral requests, is unclear.

While scribes seem to have many benefits, our study suggests they may not improve time to completion of out-of-visit tasks. This suggests that the key value of scribes may be their impact on other aspects of care, such as productivity, after-hours use of the EMR, and provider satisfaction. Reducing the time to completion for these tasks likely requires other interventions targeted to achieve those outcomes. For example, scribes could be trained to transcribe provider responses to inbox messages such as patient messages. Care redesign efforts have historically focused on redesigning visit workflows to engage care team members. Increasingly, these care redesign efforts are focusing on reducing the burden of the EMR on providers by engaging team members in out-of-visit tasks such as in-basket management.25

View this table:
[Table4](http://www.jabfm.org/content/34/1/70/T4)

Appendix Table 1. Characteristics of Scribed and Nonscribed Providers

## Notes

*   This article was externally peer reviewed.

*   *Funding:* This study was funded by institutional funds of the Cambridge Health Alliance and the Department of Family Medicine at Tufts University School of Medicine.

*   *Conflict of interest:* None.

*   To see this article online, please go to: [http://jabfm.org/content/34/1/70.full](http://jabfm.org/content/34/1/70.full).

*   Received for publication June 30, 2020.
*   Revision received August 31, 2020.
*   Accepted for publication September 10, 2020.

## References

1.  1.Dexter EN, Fields S, Rdesinski RE, Sachdeva B, Yamashita D, Marino M. Patient-provider communication: does electronic messaging reduce incoming telephone calls? J Am Board Fam Med 2016;29:613–9.
    
    [Abstract/FREE Full Text](http://www.jabfm.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NToiamFiZnAiO3M6NToicmVzaWQiO3M6ODoiMjkvNS82MTMiO3M6NDoiYXRvbSI7czoxOToiL2phYmZwLzM0LzEvNzAuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

2.  2.Murphy DR, Reis B, Sittig DF, Singh H. Notifications received by primary care practitioners in electronic health records: a taxonomy and time analysis. Am J Med 2012;125:209.e1–209.e7.
    
    [CrossRef](http://www.jabfm.org/lookup/external-ref?access_num=10.1016/j.amjmed.2011.07.029&link_type=DOI) 
    
    [PubMed](http://www.jabfm.org/lookup/external-ref?access_num=22269625&link_type=MED&atom=%2Fjabfp%2F34%2F1%2F70.atom) 

3.  3.Baron RJ. What’s keeping us so busy in primary care? A snapshot from one practice. N Engl J Med 2010;362:1632–6.
    
    [CrossRef](http://www.jabfm.org/lookup/external-ref?access_num=10.1056/NEJMon0910793&link_type=DOI) 
    
    [PubMed](http://www.jabfm.org/lookup/external-ref?access_num=20427812&link_type=MED&atom=%2Fjabfp%2F34%2F1%2F70.atom) 
    
    [Web of Science](http://www.jabfm.org/lookup/external-ref?access_num=000277117400015&link_type=ISI) 

4.  4.Arndt BG, Beasley JW, Watkinson MD, et al. Tethered to the EHR: primary care physician workload assessment using EHR event log data and time-motion observations. Ann Fam Med 2017;15:419–26.
    
    [Abstract/FREE Full Text](http://www.jabfm.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6ODoiYW5uYWxzZm0iO3M6NToicmVzaWQiO3M6ODoiMTUvNS80MTkiO3M6NDoiYXRvbSI7czoxOToiL2phYmZwLzM0LzEvNzAuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

5.  5.Lieu TA, Altschuler A, Weiner JZ, et al. Primary care physicians’ experiences with and strategies for managing electronic messages. JAMA Netw Open 2019;2:e1918287.
    
    

6.  6.Murphy DR, Meyer AND, Russo E, Sittig DF, Wei L, Singh H. The burden of inbox notifications in commercial electronic health records. JAMA Intern Med 2016;176:559–60.
    
    

7.  7.Sinsky C, Colligan L, Li L, et al. Allocation of physician time in ambulatory practice: a time and motion study in 4 specialties. Ann Intern Med 2016;165:753–60.
    
    [CrossRef](http://www.jabfm.org/lookup/external-ref?access_num=10.7326/M16-0961&link_type=DOI) 
    
    [PubMed](http://www.jabfm.org/lookup/external-ref?access_num=27595430&link_type=MED&atom=%2Fjabfp%2F34%2F1%2F70.atom) 

8.  8.Shanafelt TD, Hasan O, Dyrbye LN, et al. Changes in burnout and satisfaction with work-life balance in physicians and the general us working population between 2011 and 2014. Mayo Clin Proc 2015;90:1600–13.
    
    [CrossRef](http://www.jabfm.org/lookup/external-ref?access_num=10.1016/j.mayocp.2015.08.023&link_type=DOI) 
    
    [PubMed](http://www.jabfm.org/lookup/external-ref?access_num=26653297&link_type=MED&atom=%2Fjabfp%2F34%2F1%2F70.atom) 

9.  9.Willard-Grace R, Knox M, Huang B, Hammer H, Kivlahan C, Grumbach K. Burnout and health care workforce turnover. Ann Fam Med 2019;17:36–41.
    
    [Abstract/FREE Full Text](http://www.jabfm.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6ODoiYW5uYWxzZm0iO3M6NToicmVzaWQiO3M6NzoiMTcvMS8zNiI7czo0OiJhdG9tIjtzOjE5OiIvamFiZnAvMzQvMS83MC5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

10. 10.Østbye T, Yarnall KSH, Krause KM, Pollak KI, Gradison M, Michener JL. Is there time for management of patients with chronic diseases in primary care? Ann Fam Med 2005;3:209–14.
    
    [Abstract/FREE Full Text](http://www.jabfm.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6ODoiYW5uYWxzZm0iO3M6NToicmVzaWQiO3M6NzoiMy8zLzIwOSI7czo0OiJhdG9tIjtzOjE5OiIvamFiZnAvMzQvMS83MC5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

11. 11.Gottschalk A, Flocke SA. Time spent in face-to-face patient care and work outside the examination room. Ann Fam Med 2005;3:488–93.
    
    [Abstract/FREE Full Text](http://www.jabfm.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6ODoiYW5uYWxzZm0iO3M6NToicmVzaWQiO3M6NzoiMy82LzQ4OCI7czo0OiJhdG9tIjtzOjE5OiIvamFiZnAvMzQvMS83MC5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

12. 12.Gidwani R, Nguyen C, Kofoed A, et al. Impact of scribes on physician satisfaction, patient satisfaction, and charting efficiency: a randomized controlled trial. Ann Fam Med 2017;15:427–33.
    
    [Abstract/FREE Full Text](http://www.jabfm.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6ODoiYW5uYWxzZm0iO3M6NToicmVzaWQiO3M6ODoiMTUvNS80MjciO3M6NDoiYXRvbSI7czoxOToiL2phYmZwLzM0LzEvNzAuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

13. 13.Addesso LC, Nimmer M, Visotcky A, Fraser R, Brousseau DC. Impact of medical scribes on provider efficiency in the pediatric emergency department. Acad Emerg Med 2018;26:174–82.
    
    

14. 14.Imdieke BH, Martel ML. Integration of medical scribes in the primary care setting: improving satisfaction. J Ambul Care Manage 2017;40:17–25.
    
    

15. 15.Zallman L, Finnegan K, Roll D, Todaro M, Oneiz R, Sayah A. Impact of medical scribes in primary care on productivity, face-to-face time, and patient comfort. J Am Board Fam Med J Am Board Fam Med 2018;31:612–9.
    
    

16. 16.Earls ST, Savageau JA, Begley S, Saver BG, Sullivan K, Chuman A. Can scribes boost FPs’ efficiency and job satisfaction? J Fam Pract 2017;66:206–14.
    
    

17. 17.Pozdnyakova A, Laiteerapong N, Volerman A, et al. Impact of medical scribes on physician and patient satisfaction in primary care. J Gen Intern Med 2018;33:1109–15.
    
    

18. 18.Mishra P, Kiang JC, Grant RW. Association of medical scribes in primary care with physician workflow and patient experience. JAMA Intern Med 2018;178:1467–72.
    
    

19. 19.Wade-Vuturo AE, Mayberry LS, Osborn CY. Secure messaging and diabetes management: experiences and perspectives of patient portal users. J Am Med Inform Assoc 2013;20:519–25.
    
    [CrossRef](http://www.jabfm.org/lookup/external-ref?access_num=10.1136/amiajnl-2012-001253&link_type=DOI) 
    
    [PubMed](http://www.jabfm.org/lookup/external-ref?access_num=23242764&link_type=MED&atom=%2Fjabfp%2F34%2F1%2F70.atom) 

20. 20.Ralston JD, Carrell D, Reid R, Anderson M, Moran M, Hereford J. Patient web services integrated with a shared medical record: patient use and satisfaction. J Am Med Inform Assoc 2007;14:798–806.
    
    [CrossRef](http://www.jabfm.org/lookup/external-ref?access_num=10.1197/jamia.M2302&link_type=DOI) 
    
    [PubMed](http://www.jabfm.org/lookup/external-ref?access_num=17712090&link_type=MED&atom=%2Fjabfp%2F34%2F1%2F70.atom) 

21. 21.Hefner JL, MacEwan SR, Biltz A, Sieck CJ. Patient portal messaging for care coordination: a qualitative study of perspectives of experienced users with chronic conditions. BMC Fam Pract 2019;20:57.
    
    

22. 22.Murphy DR, Satterly T, Giardina TD, Sittig DF, Singh H. Practicing clinicians’ recommendations to reduce burden from the electronic health record inbox: a mixed-methods study. J Gen Intern Med 2019;34:1825–32.
    
    

23. 23.Tukey JW. Exploratory data analysis, EDA. Menlo Park, CA: Addison-Wesley; 1977.
    
    

24. 24.DeChant PF, Acs A, Rhee KB, et al. Effect of organization-directed workplace interventions on physician burnout: a systematic review. Mayo Clin Proc Innov Qual Outcomes 2019;3:384–408.
    
    

25. 25.Smith PC, Lyon C, English AF, Conry C. Practice transformation under the University of Colorado’s primary care redesign model. Ann Fam Med 2019;17:S24–S32.
    
    [Abstract/FREE Full Text](http://www.jabfm.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6ODoiYW5uYWxzZm0iO3M6NToicmVzaWQiO3M6MTQ6IjE3L1N1cHBsXzEvUzI0IjtzOjQ6ImF0b20iO3M6MTk6Ii9qYWJmcC8zNC8xLzcwLmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==)