Points of Concordance, Points of Discordance: A Qualitative Examination of Telemedicine Implementation
======================================================================================================

* Tristen L. Hall
* Lauri Connelly
* Elizabeth W. Staton
* Jodi Summers Holtrop
* Amber Sieja
* Kyle Knierim
* Heather Holmstrom

## Abstract

*Introduction:* Health systems undertook a rapid transition to increase the use of telemedicine in the wake of the COVID-19 pandemic. A continued need for telemedicine services in the coming years is likely. This article examines telemedicine from multiple stakeholders' perspectives considering reach, effectiveness, adoption, implementation, and maintenance (RE-AIM) outcomes.

*Methods:* Semistructured interviews were conducted with primary care practice team members and patients. Rapid qualitative analysis was used to identify themes in experiences and perceptions related to telemedicine implementation. The RE-AIM implementation framework was applied to thematic findings to understand influences on implementation outcomes.

*Results:* Twenty-four practice members and 17 patients across 5 clinics participated. All stakeholder groups reported that technological capabilities influenced patients' access to telemedicine and that certain patients and reasons for visits were not appropriate for telemedicine. All groups felt that telemedicine was a good option for some patients some of the time but not all patients all of the time.

*Discussion:* Telemedicine works well if it is used for the appropriate visits and patient types and with needed technological elements. Older age may limit the feasibility of telehealth for some patients. Added administrative work and associated costs support systematic screening to determine visit appropriateness for telemedicine.

*   Information Technology
*   Nonverbal Communication
*   Primary Health Care
*   Qualitative Methods
*   Telemedicine

## Introduction

Many health systems and practices across the world undertook a rapid transition to increase the use of telemedicine care in the wake of the COVID-19 pandemic.1⇓–3 Telemedicine, also known as virtual care, refers to the provision of health care and related services such as education and information through modes of telecommunication, including virtual and phone visits.4 Quarantine restrictions and social distancing strategies designed to minimize spread of the novel coronavirus reduced many primary care and other practices to almost entirely telemedicine care, rather than in-person, clinic-based care delivery, for a period of time during 2020.5 Practices' and hospitals' capacity and readiness to deliver telemedicine services have historically varied, with geographic differences in level of telemedicine adoption and less than half of primary care physicians offering telemedicine services as of 2016.6,7 However, primary care telemedicine services increased substantially, with surveys showing a majority (69%) of primary care clinicians motivated to use telemedicine to meet patient demand during the pandemic.8

Although telemedicine arguably became necessary during the early part of COVID-19, it is likely here to stay, with physicians and major professional organizations calling for its expanded use beyond COVID-19.9⇓–11 Learning more about telemedicine implementation could be valuable for improving primary care delivery. Certain types of visits, such as managing mental health concerns or visits that do not necessitate a physical examination, may be more amenable to telemedicine than others.12,13 Findings on patient satisfaction and quality of care for telemedicine compared with in-person visits are inconsistent: while some studies have found that patients can be equally or more satisfied with telemedicine,14 others have found that patients may rate certain aspects of telemedicine care as being of lower quality than in-person care.12 Further, clinicians may find telemedicine to be a less effective tool for managing certain patient concerns12 and understandably may be hesitant to deliver clinical care without a physical examination or lab testing.13 In contrast, systematic reviews have found that the quality of telemedicine care can equal that of in-person visits for certain types of care, including mental health, dermatology, and rehabilitation.15 Despite promising findings on quality of care and satisfaction, disparities in telemedicine use among patients remain. Younger adults tend to be more likely to use telemedicine than older adults.12,16,17 Black and Hispanic patients are less likely to use telemedicine in comparison to White and Asian patients, and Spanish-speaking patients are less likely than English-speaking patients to have a telemedicine visit.17 There is lower use of telemedicine in rural areas and in lower income communities, and technological barriers such as a poor Internet signal can still hinder the virtual connection between clinicians and patients.2,12

As a continued need for telemedicine services is likely, there is a need to understand the factors that facilitate or hinder implementation of effective telemedicine care. RE-AIM (reach, effectiveness, adoption, implementation, and maintenance) is an implementation framework that can offer insight as to the factors that influence the outcomes (or domains) of intervention reach and effectiveness to patients, and the adoption, implementation, and maintenance of interventions by providers and settings to affect those patient outcomes.18,19 It is the most widely used implementation science framework in the field and offers unique insight by breaking apart aspects of implementation that can be challenges.18 We share the results of an analysis that examines RE-AIM by the perspective of different stakeholders such that we can identify places where there are similarities (concordance) and differences (discordance). The purpose of this article is to enhance understanding about how telemedicine can be most effectively implemented in primary care by examining it from the viewpoints of key stakeholders through the lens of factors affecting RE-AIM outcomes.

## Methods

### Setting

This study was reviewed and approved as exempt by the Colorado Multiple Institutional Review Board. The study involved 5 family medicine clinics within 30 miles of Denver, CO, that are part of a large academic health system. The cities across this region differ in size and demographics, resulting in greater variation in the sample than using 1 clinic in 1 city. Table 1 describes the practices. Clinic 1 is a training site and thus has more clinicians and behavioral health clinicians than other participating sites.

View this table:
[Table 1.](http://www.jabfm.org/content/35/3/517/T1)

Table 1. 
Characteristics of Primary Care Clinics That Participated in Study Interviews

### Participant Recruitment and Description

We created groups of 4 types of study participants to gather multiple perspectives on telemedicine implementation. Three of the participant types, clinicians (MD, DO, NP, PA), clinical staff (medical assistants, nurses, and care managers), and administrative staff (reception and practice management), worked within the family medicine practices. These practice members were recruited for individual semistructured interviews through each site's medical director, who nominated 5 potential interviewees from the practice including at least 1 person from each participant type with relevant experience conducting or supporting virtual care. In this health system, telemedicine or virtual care consists of computer-based visits using both audio and video, or telephone visits using audio only if video visits were not feasible.

The fourth type of participants were patients with and without experience taking part in a telemedicine visit. We included patients to gather their perspectives regarding the experience itself or why they did not participate in virtual care. Patients from each of the 5 clinics were purposefully selected based on telemedicine visit or offered and declined. During a 2-month window, patients were approached by the practice virtual visit champion following an existing visit and invited to participate in the study. In addition, patients participating in a patient advisory committee were invited to participate by practice facilitators. Refusals were not tracked, but the number of declines was low.

### Data Collection

Data collection consisted of individual one-on-one interviews conducted by videoconference or phone. A trained qualitative researcher (TH, JH, LC) conducted the interviews with each participant using semistructured interview guides: one for practice member interviews and one for patient interviews. Interview guides were developed by the study team's qualitative researchers and reviewed and edited by the team's clinician researchers. Interviewers debriefed after each conducting 2 interviews within each stakeholder group to discuss experiences with the interview guide and make any needed revisions or additions to the interview guide. The practice member guide sought to gain insight on participants' background characteristics and RE-AIM domains and included questions on the practice member's background, including work role and responsibilities before and during COVID-19, comparisons between providing or supporting in-person care compared with telemedicine, offering telemedicine from the clinic compared with from home, factors that facilitated the transition to telemedicine, circumstances under which telemedicine did not work well, practice members' teamwork, characteristics of patients who participated in telemedicine or did not, potential concerns about patient care outcomes, concerns about the transition back to in-person care, and outstanding needs to facilitate future telemedicine services. The patient interview guide included questions about patients' history with the practice and use of the care team such as behavioral health or care managers, how telemedicine services were offered, why they considered a telemedicine visit or did not, key steps in the virtual visit, elements that worked well or needed improvement, willingness to use telemedicine in the future, and comments about their ability to relate to the clinician and the quality of the visit. Each interview was audio recorded and no incentive was provided. Interviews were conducted until it became clear through an iterative process that reasonable saturation had occurred and key concepts had emerged and were captured for respective groups.

Using rapid analysis methods,20⇓–22 the researchers took extensive typed notes (almost verbatim) during the interview. Immediately after the interview, the researcher completed a structured summary template. The template for the practice member interviews had the following key groupings: background information on participant, telemedicine experience, patient issues, transition to telemedicine, and transition to in-person care. The patient interviews had background information on participant, telemedicine experience (or lack thereof), and in-person care. Both templates also included (1) key points from the interview overall and (2) interviewer's impressions. The researcher reviewed the summary with the recording when any gaps were evident in memory. The recordings were not transcribed but kept for future reference. Interviews ranged from 30 to 60 minutes but were generally 40 to 45 minutes.

### Data Analysis

We used rapid qualitative analysis. This is an emerging but rigorous qualitative analytic method used for semistructured work when information is already known about the subject (rather than being totally exploratory) and the clinical team would benefit from receiving results faster than the long time frame of typical qualitative work.20⇓–22 It is especially useful for implementation research, such as this study.

For the practice member interviews, each of the interview summaries was gathered into a matrix (ie, table) that provided space for each practice to be a row and key summary groupings in a column. Thus, each cell represented summary information on key topical areas for each practice. There were a total of 25 cells (5 rows of practices with 5 cells per practice). Within each cell, the role type (clinician, administrative staff, or clinical staff or combinations of these roles) was noted for where the information originated. Although the summaries for each practice were a means of identifying key factors, the intent was to analyze overall across practices by role. Both researchers working on this portion of the study (JH, TH) completed their own matrix and then combined the 2 versions together through discussion by eliminating areas of duplication and then discussing areas of unique contributions until a final matrix was created. Then the researchers created a second matrix that considered results by role across practices to develop the key themes for this portion of the work.

For the patient interviews, a similar process ensued; however, the results were not divided by practice and instead by patients experiencing telemedicine or not participating in telemedicine. In this case, only 1 researcher (LC) conducted all the interviews and the summaries, and a different researcher (TH) created the matrix.

The next step of the analysis was to consider the implementation framework, RE-AIM. This process generally followed a joint display table analysis as described in Guetterman et al.23 We created a table (Table 2) to organize the thematic responses by RE-AIM domain and role type including patients. The final column summarized the analysis of the themes across the role types for that RE-AIM domain. The researchers (TH, JH) individually and then collectively created their own completed table using this format and then through ongoing discussion reconciled and summarized the cell thematic information and the extent to which there was concordance/discordance. To confirm the correct categorization of responses in RE-AIM domains, the table was discussed with Dr. Russell Glasgow, RE-AIM developer,18,19,24 until the appropriate classification was determined. In addition, 1 coauthor (JH) has experience using RE-AIM for analysis of qualitative data.25

View this table:
[Table 2.](http://www.jabfm.org/content/35/3/517/T2)

Table 2. 
Factors by RE-AIM Dimension That Affected Telemedicine Use and Usefulness: Concordance and Discordance by Participant Role

The final step in the analysis was descriptive. A key theme that emerged across all the groups was that telemedicine works when it is the right “fit” including the right patient, right clinician, right support resources, and right type of visit. Thus, the researchers examined the responses across participant types and organized the themes around these “right fit” descriptions. Then they described the summary information narratively and cross-checked it with selected practice members (ie, member checking) and with coinvestigators.

## Results

### Study Participants

There were a total of 41 participants across the 5 clinics. Table 1 provides a description of the practices. Participants included 6 clinicians, 11 administrative staff, 7 clinical staff, and 17 patients. Of the patients, 14 received care through telemedicine between March and September 2020, and 3 were offered but did not. Participants had telemedicine visits for primary complaints such as skin lesion, ankle sprain, pain, gastrointestinal issues, mental health, medication management, fluid buildup following a previous heart issue, and vertigo. Of the 14 telemedicine participants, 6 had also experienced an in-person visit during this time period between the onset of COVID-19 restrictions and data collection. Participants with telemedicine experience included 9 women and 5 men ranging in age from mid-30s to mid-80s. The 3 patients without a telemedicine visit were male and older than 60.

### Telehealth as an Implementation Process

Table 2 outlines stakeholder views across the RE-AIM domains. Overall, there were similarities across groups regarding who was reached by telemedicine (reach), whether it was effective, and what it took to adopt and implement it well. All stakeholder groups reported that technological capabilities influenced the extent to which patients had access to telemedicine visits (reach), because some patients did not have adequate Internet service, video capabilities, or capability to use the technology required for telemedicine as it was deployed in this health system. Clinic members described how this led to administrative staff spending a great deal of time working with patients to troubleshoot technological challenges, particularly in the early stage of telemedicine use. When video aspects of virtual visits did not function due to limitations of technology used, telemedicine visits were converted to telephone using voice only, limiting the quality of communication during virtual visits but expanding the reach. Some clinicians perceived that older patients or those who were less tech-savvy in general had more difficulty with telemedicine or were less likely in general to try it, though older patients we interviewed noted few such challenges. Patients highlighted safety as a motivator to take part in telemedicine, while other groups did not emphasize this. Overall, patients and clinicians had more comments about aspects of reach than other stakeholders.

There was agreement between patients, clinicians, and clinical staff that certain visits, particularly those that required a physical examination, were not appropriate for telemedicine (effectiveness). Similarly, patients, clinicians, and staff recognized that not all patients are appropriate for telemedicine—some lack required technology access or knowledge, or cognitive abilities that limit effective communication about physical symptoms; some simply prefer not to participate in virtual health care. Patients and clinical staff highlighted how limited nonverbal communication over video or phone affected visit effectiveness. Further, patients hypothesized that some clinicians' communication and listening skills may be less amenable to telemedicine, though none experienced this during the virtual visits discussed for this study. However, neither clinicians nor administrative staff highlighted communication as an influence on the effectiveness of telemedicine. Patients noted that telemedicine and in-person visits were of similar quality level more so than did practice member groups.

Clinicians and staff noted difficulty with remote communication or management across team members, with some struggles with team members' responsiveness (implementation). In contrast, fewer clinicians noted differences in the process and experience of providing telemedicine in comparison to in-person care than did clinical and administrative staff. Both patients and clinic members suggested that previsit steps to ensure the visit type was appropriate and optimized for telemedicine (eg, length of visit) would benefit the process. All participant groups shared similar comments about their hope for continued availability of telemedicine moving forward past COVID-19 (maintenance) when the right conditions were in place (see “right fit” section below), with administrators and clinicians recognizing the need for continued payment and approval for use by insurers and regulators. Patients had less to comment on regarding clinician and team adoption of telemedicine but did note areas that might influence adoption. For instance, patients recognized that practice teams might be especially willing to adopt telemedicine when the workforce was busy with a heavy workload. They suggested that telemedicine might help with efficiency during these times.

In summary, patients and clinicians shared similar feedback on the telemedicine experience, speaking from both sides of the visit experience. Patients were concerned about the quality of communication with clinicians, and clinicians worried about providing a quality visit for the patient. The staff members, either administrative or clinical, had more logistic and practical issues with the determination of how it worked from home and if needed resources were available. All groups felt that telemedicine was a good option for some patients, some of the time.

### “Right Fit” Themes

Throughout analysis, it emerged that telemedicine works when the right set of conditions are in place. We chose to call this the “right fit” situation. If any one of these conditions was not met, the telemedicine visit would not happen or not happen well, affecting effectiveness and implementation especially. Table 3 lists different factors that are associated with this right fit situation. Three main factors rose to the top: technology, mindset, and health issue. Some minimum technological capabilities were needed by both patients and clinics, such as equipment and Internet bandwidth, though some visits were converted to telephone with audio only if one or more of these elements were missing. Mindset consisted of knowledge of how to do a telemedicine visit and comfort with doing so. Practice teams needed appropriate training and preparation to deliver telemedicine, and patients had to be willing to engage in it, particularly when visits that were originally scheduled to be in person were converted to telemedicine out of precaution for COVID-19. The health condition addressed in the visit had to be appropriate to be conducted without a physical examination, or at least with a modified physical examination through video examination and questioning the patient.

View this table:
[Table 3.](http://www.jabfm.org/content/35/3/517/T3)

Table 3. 
Factors Associated with the “Right Fit” for Use of Telemedicine

## Discussion

Telemedicine is likely to work well if it is used for the appropriate visits and patient types and with needed technologies. By exploring telemedicine using the RE-AIM implementation science framework, we illustrated that telemedicine is perceived as a valuable service when applied to the right clinical situation and by a prepared care team. For instance, previous studies indicate that several chronic conditions are at least sometimes amenable to telemedicine in primary care. Studies further suggest that between one third and three quarters of primary care visits could be delivered by telemedicine.7,26

Our findings reinforce those of other studies, but this study is unique in that it adds detail from multiple perspectives, including patients, and how they compare with one another. This approach exposes implementation challenges that are not evident in other studies. For example, previous studies indicate that clinicians and staff must be motivated and interested in telemedicine for implementation to be successful.27 Our study adds that implementation also hinges to some extent on a patient's motivation or mindset—some patients are simply not open to participating in virtual care, and this must be considered in the wider implementation of telemedicine and delineation of which types of visits are most appropriate to hold virtually. This preference is separate from the ability to use technology for virtual care. Flexibility to offer in-person visits regardless of visit type should be maintained when it is safe to do so.

Practice members and patients both had concerns about optimizing telemedicine visits and hope for the option of telemedicine or in-person visits in the future. Patients, clinicians, and clinic staff were supportive of having telemedicine as an option in the future, after COVID-19 precautions have passed. Our findings are consistent with evidence that physicians feel telemedicine can improve access to care as a result of its convenience, and our study adds similar input on this topic from multiple stakeholders, including patients, while also exposing areas that, if not addressed, could lessen the adoption or effectiveness of telemedicine.28 At the same time, we found that patients understand the limitations of telemedicine, such as when its use is not appropriate or feasible. This supports attention to identified areas of improvement for its continued use, such as adequate previsit planning and further distinguishing which visit types are best for telemedicine. It also shows that patients recognize the benefit of telemedicine not only to themselves but potentially to the well-being of clinicians through reduced stress.

There were some inconsistencies between practice members' and patients' perceptions of telemedicine reach, effectiveness, and adoption. However, most differences were found within groups rather than across groups. An example within groups was that some clinicians and staff were able to implement telemedicine effectively from home while some were not. An example from across groups was that with the exception of communication with clinicians, patients reported telemedicine being of similar quality to in-person visits more than did practice members.

Consistent with the perceptions of practice members but not patients in this study, most evidence indicates that older patients use telemedicine, particularly video options, less often than other age groups.12,16,17,29,30 Further, primary care clinicians have concerns about the ability of some patients to effectively participate in telemedicine, due to lack of needed technology or related capabilities.28 This, in addition to patients' thoughts that telemedicine is less appropriate or feasible for those with limited dexterity or cognitive abilities, supports efforts to systematically develop strategies for older adults to participate as technology advances.31 Older adults are more likely than other groups of patients to experience these limitations, and research is needed to prevent exacerbating the digital divide.31 To address this, some have even called for insurance providers to cover Internet-enabled tablets to support access and help address the growing digital divide.32 However, uniquely, we found evidence of patients who were older who did enjoy and use telemedicine without technical challenges.

Building on the need to better understand how team members can support clinicians in a model combining telemedicine and in-person care delivery,26 we found that clinical and administrative staff are crucial to ensuring that telemedicine visits run smoothly. Clinic members emphasized the extra administrative work and coordination required to set up virtual visits, particularly early in the pandemic, which may be because telemedicine requires more telephone calls and e-mail messages with the patient to arrange care and answer questions. This has financial implications for practices because much of this work outside of patient visits is not covered by reimbursement,33 highlighting the need for more efficient processes that ensure visits are appropriate and optimized for telemedicine. Much like how staff prepare for patient visits by cleaning and stocking examination rooms, collecting medical and social history from patients, and conducting screenings before the clinician enters the examination room, staff troubleshot technical issues with patients, checked patients in, ensured needed forms were completed, and walked patients through the telemedicine process before these visits. To support staff engagement in virtual care, it may help to frame these tasks as analogous to those completed before and during in-person visits, rather than viewing it as extra work.

### Limitations

These results were gathered from numerous clinics, but they were all only primary care and within 1 health system that prioritized video-based telemedicine encounters over other telemedicine modalities. Thus, our findings may not represent other geographic regions, other practices, or other telemedicine modalities. We have presented how we believe the themes we identified in this article resonate with the findings of others; however, differences are likely to still exist. As with all qualitative methods, the sampling is purposeful rather than representative and not meant to stand for a large sample size. We intended to meaningfully describe the key issues and how they were experienced by the participants, thus adding to the richness of understanding regarding this evolving topic. This study included limited representation from patients who had not experienced a telemedicine visit or were resistant to participating in telemedicine, which may have skewed findings to include more positive perceptions of telemedicine than would be found in the general population. In addition, since patients were recruited by the practice members, they have been biased toward being more positive about telemedicine, even though we did not find that to be the case. Clinicians, staff, and administrators were also purposefully identified by medical directors, which may have resulted in bias as interviewees may have been selected for reasons that are unclear. We caution that the methods used only point us to what we might consider a potential set of circumstances that may be important for telemedicine implementation. Further work will be needed to test out these findings.

### Recommendations

Consistent with previous research, we found that certain types of visits, patients, and other conditions are more amenable to telemedicine than others, though further research is needed to identify more specifically what combination of factors is needed for optimal implementation. A specific suggestion for the future might include development of a tool to assess the “right fit” dimensions at the point of scheduling. Another suggestion is that, given how patients voiced concerns regarding communication gaps yet clinicians did not express that concern, training about this new way of conducting a visit could improve the patient experience.

## Notes

*   This article was externally peer reviewed.

*   *Funding:* The authors report no funding support for this study.

*   *Conflict of interest:* The authors declare no conflicting or competing interests to report.

*   To see this article online, please go to: [http://jabfm.org/content/35/3/517.full](http://jabfm.org/content/35/3/517.full).

*   Received for publication August 4, 2021.
*   Revision received October 19, 2021.
*   Revision received November 30, 2021.
*   Accepted for publication December 9, 2021.

## References

1.  1.Fisk M, Livingstone A, Pit SW. Telehealth in the context of COVID-19: changing perspectives in Australia, the United Kingdom, and the United States. J Med Internet Res 2020;22:e19264.
    
    [CrossRef](http://www.jabfm.org/lookup/external-ref?access_num=10.2196/19264&link_type=DOI) 
    
    [PubMed](http://www.jabfm.org/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fjabfp%2F35%2F3%2F517.atom) 

2.  2.Patel SY, Mehrotra A, Huskamp HA, Uscher-Pines L, Ganguli I, Barnett ML. Variation in telemedicine use and outpatient care during the COVID-19 pandemic in the United States. Health Aff (Millwood) 2021;40:349–58.
    
    

3.  3.Knierim K, Palmer C, Kramer ES, et al. Lessons learned during COVID-19 that can move telehealth in primary care forward. J Am Board Fam Med 2021;34:S196–S202.
    
    [Abstract/FREE Full Text](http://www.jabfm.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NToiamFiZnAiO3M6NToicmVzaWQiO3M6MTg6IjM0L1N1cHBsZW1lbnQvUzE5NiI7czo0OiJhdG9tIjtzOjIwOiIvamFiZnAvMzUvMy81MTcuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

4.  4.NEJM Catalyst [Internet]. What is telehealth? N Engl J Med Catal; 2018. Available from: [https://catalyst.nejm.org/doi/full/10.1056/CAT.18.0268](https://catalyst.nejm.org/doi/full/10.1056/CAT.18.0268).
    
    

5.  5.Monaghesh E, Hajizadeh A. The role of telehealth during COVID-19 outbreak: a systematic review based on current evidence. BMC Public Health 2020;20:1193.
    
    [CrossRef](http://www.jabfm.org/lookup/external-ref?access_num=10.1186/s12889-020-09301-4&link_type=DOI) 
    
    [PubMed](http://www.jabfm.org/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fjabfp%2F35%2F3%2F517.atom) 

6.  6.Hong Y-R, Lawrence J, Williams D Jr., Mainousi A III.. Population-level interest and telehealth capacity of US hospitals in response to COVID-19: cross-sectional analysis of Google search and national hospital survey data. JMIR Public Health Surveill 2020;6:e18961.
    
    

7.  7.Jetty A, Jabbarpour Y, Westfall M, Kamerow DB, Petterson S, Westfall JM. Capacity of primary care to deliver telehealth in the United States. J Am Board Fam Med 2021;34;S48–S54.
    
    [Abstract/FREE Full Text](http://www.jabfm.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NToiamFiZnAiO3M6NToicmVzaWQiO3M6MTc6IjM0L1N1cHBsZW1lbnQvUzQ4IjtzOjQ6ImF0b20iO3M6MjA6Ii9qYWJmcC8zNS8zLzUxNy5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

8.  8.Larry A. Green Center [Internet]. Quick COVID-19 Primary Care Survey; 2020. Available from: [https://www.green-center.org/covid-survey](https://www.green-center.org/covid-survey).
    
    

9.  9.AAFP [Internet]. AAFP advocacy focus: telehealth and telemedicine; 2021 [cited 2021 Nov 29] Available from: [https://www.aafp.org/advocacy/advocacy-topics/health-it/telehealth-telemedicine.html](https://www.aafp.org/advocacy/advocacy-topics/health-it/telehealth-telemedicine.html).
    
    

10. 10.Thomson K [Internet]. Congress: act now to ensure telehealth access for Medicare beneficiaries. Health Affairs Blog; 2021. Available from: [https://www.healthaffairs.org/do/10.1377/forefront.20210505.751442/full/](https://www.healthaffairs.org/do/10.1377/forefront.20210505.751442/full/).
    
    

11. 11.North S. These four telehealth changes should stay, even after the pandemic. FPM 2021;28:9–11.
    
    

12. 12.Hammersley V, Donaghy E, Parker R, et al. Comparing the content and quality of video, telephone, and face-to-face consultations: a non-randomised, quasi-experimental, exploratory study in UK primary care. Br J Gen Pract 2019;69:e595–e604.
    
    [Abstract/FREE Full Text](http://www.jabfm.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NDoiYmpncCI7czo1OiJyZXNpZCI7czoxMToiNjkvNjg2L2U1OTUiO3M6NDoiYXRvbSI7czoyMDoiL2phYmZwLzM1LzMvNTE3LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

13. 13.Barney A, Buckelew S, Mesheriakova V, Raymond-Flesch M. The COVID-19 pandemic and rapid implementation of adolescent and young adult telemedicine: challenges and opportunities for innovation. J Adolesc Health 2020;37:164–71.
    
    

14. 14.Ramaswamy A, Yu M, Drangsholt S, et al. Patient satisfaction with telemedicine during the COVID-19 pandemic: retrospective cohort study. J Med Internet Res 2020;22:e20786.
    
    

15. 15.Shigekawa E, Fix M, Corbett G, Roby DH, Coffman J. The current state of telehealth evidence: a rapid review. Health Aff (Millwood) 2018;37:1975–82.
    
    [CrossRef](http://www.jabfm.org/lookup/external-ref?access_num=10.1377/hlthaff.2018.05132&link_type=DOI) 
    
    [PubMed](http://www.jabfm.org/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fjabfp%2F35%2F3%2F517.atom) 

16. 16.Mann DM, Chen J, Chunara R, Testa PA, Nov O. COVID-19 transforms health care through telemedicine: evidence from the field. J Am Med Inform Assoc 2020;27:1132–5.
    
    [CrossRef](http://www.jabfm.org/lookup/external-ref?access_num=10.1093/jamia/ocaa072&link_type=DOI) 
    
    [PubMed](http://www.jabfm.org/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fjabfp%2F35%2F3%2F517.atom) 

17. 17.Rodriguez JA, Betancourt JR, Sequist TD, Ganguli I. Differences in the use of telephone and video telemedicine visits during the COVID-19 pandemic. Am J Manag Care 2021;27:21–6.
    
    

18. 18.Glasgow RE, Harden SM, Gaglio B, et al. RE-AIM planning and evaluation framework: adapting to new science and practice with a 20-year review. Front Public Health 2019;7:64.
    
    [PubMed](http://www.jabfm.org/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fjabfp%2F35%2F3%2F517.atom) 

19. 19.RE-AIM [Internet]. Welcome to RE-AIM and PRISM: implementation in context; 2021 [cited 2021 June 16]. Available from: [https://www.re-aim.org/](https://www.re-aim.org/).
    
    

20. 20.Taylor B, Henshall C, Kenyon S, Litchfield I, Greenfield S. Can rapid approaches to qualitative analysis deliver timely, valid findings to clinical leaders? A mixed methods study comparing rapid and thematic analysis. BMJ Open 2018;8:e019993.
    
    [Abstract/FREE Full Text](http://www.jabfm.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NzoiYm1qb3BlbiI7czo1OiJyZXNpZCI7czoxMjoiOC8xMC9lMDE5OTkzIjtzOjQ6ImF0b20iO3M6MjA6Ii9qYWJmcC8zNS8zLzUxNy5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

21. 21.Palinkas LA, Mendon SJ, Hamilton AB. Innovations in mixed methods evaluations. Annu Rev Public Health 2019;40:423–42.
    
    [CrossRef](http://www.jabfm.org/lookup/external-ref?access_num=10.1146/annurev-publhealth-040218-044215&link_type=DOI) 
    
    [PubMed](http://www.jabfm.org/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fjabfp%2F35%2F3%2F517.atom) 

22. 22.Nevedal AL, Reardon CM, Opra Widerquist MA, et al. Rapid versus traditional qualitative analysis using the Consolidated Framework for Implementation Research (CFIR). Implement Sci 2021;16:67.
    
    

23. 23.Guetterman TC, Fetters MD, Creswell JW. Integrating quantitative and qualitative results in health science mixed methods research through joint displays. Ann Fam Med 2015;13:554–61.
    
    [Abstract/FREE Full Text](http://www.jabfm.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6ODoiYW5uYWxzZm0iO3M6NToicmVzaWQiO3M6ODoiMTMvNi81NTQiO3M6NDoiYXRvbSI7czoyMDoiL2phYmZwLzM1LzMvNTE3LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

24. 24.Kwan BM, McGinnes HL, Ory MG, Estabrooks PA, Waxmonsky JA, Glasgow RE. RE-AIM in the real world: use of the RE-AIM framework for program planning and evaluation in clinical and community settings. Front Public Health 2019;7:345.
    
    

25. 25.Holtrop JS, Rabin BA, Glasgow RE. Qualitative approaches to use of the RE-AIM framework: rationale and methods. BMC Health Serv Res 2018;18:177.
    
    [PubMed](http://www.jabfm.org/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fjabfp%2F35%2F3%2F517.atom) 

26. 26.Jabbarpour Y, Jetty A, Westfall M, Westfall J. Not telehealth: which primary care visits need in-person care? J Am Board Fam Med 2021;34:S162–S169.
    
    [Abstract/FREE Full Text](http://www.jabfm.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NToiamFiZnAiO3M6NToicmVzaWQiO3M6MTg6IjM0L1N1cHBsZW1lbnQvUzE2MiI7czo0OiJhdG9tIjtzOjIwOiIvamFiZnAvMzUvMy81MTcuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

27. 27.Gagnon M-P, Duplantie J, Fortin J-P, Landry R. Implementing telehealth to support medical practice in rural/remote regions: what are the conditions for success? Implement Sci 2006;1:18.
    
    [CrossRef](http://www.jabfm.org/lookup/external-ref?access_num=10.1186/1748-5908-1-18&link_type=DOI) 
    
    [PubMed](http://www.jabfm.org/lookup/external-ref?access_num=16930484&link_type=MED&atom=%2Fjabfp%2F35%2F3%2F517.atom) 

28. 28.Gomez T, Anaya YB, Shih KJ, Tarn DM. A qualitative study of primary care physicians' experiences with telemedicine during COVID-19. J Am Board Fam Med 2021;34:S61–S70.
    
    [Abstract/FREE Full Text](http://www.jabfm.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NToiamFiZnAiO3M6NToicmVzaWQiO3M6MTc6IjM0L1N1cHBsZW1lbnQvUzYxIjtzOjQ6ImF0b20iO3M6MjA6Ii9qYWJmcC8zNS8zLzUxNy5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

29. 29.Eberly LA, Kallan MJ, Julien HM, et al. Patient characteristics associated with telemedicine access for primary and specialty ambulatory care during the COVID-19 pandemic. JAMA Netw Open 2020;3:e2031640.
    
    

30. 30.Reed ME, Huang J, Graetz I, et al. Patient characteristics associated with choosing a telemedicine visit vs office visit with the same primary care clinicians. JAMA Netw Open 2020;3:e205873.
    
    

31. 31.Krishnaswami A, Beavers C, Dorsch MP, et al. Gerotechnology for older adults with cardiovascular diseases: JACC state-of-the-art review. J Am Coll Cardiol 2020;76:2650–70.
    
    

32. 32.Rajasekaran K. Access to telemedicine—are we doing all that we can during the COVID-19 pandemic? Otolaryngol Head Neck Surg 2020;163:104–6.
    
    

33. 33.Tarn DM, Hintz C, Mendez-Hernandez E, Sawlani SP, Bholat MA. Using virtual visits to care for primary care patients with COVID-19 symptoms. J Am Board Fam Med 2021;34:S147–S151.
    
    [Abstract/FREE Full Text](http://www.jabfm.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NToiamFiZnAiO3M6NToicmVzaWQiO3M6MTg6IjM0L1N1cHBsZW1lbnQvUzE0NyI7czo0OiJhdG9tIjtzOjIwOiIvamFiZnAvMzUvMy81MTcuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9)