Research ArticleOriginal Research

Using the Quadruple Aim Framework to Measure Impact of Heath Technology Implementation: A Case Study of eConsult

Clare Liddy and Erin Keely
The Journal of the American Board of Family Medicine May 2018, 31 (3) 445-455; DOI: https://doi.org/10.3122/jabfm.2018.03.170397

Abstract

Background: Health technology solutions are too often implemented without a true understanding of the system-level problem they seek to address, resulting in excessive costs, poor adoption, ineffectiveness, and ultimately failure. Before implementing or adopting health care innovations, stakeholders should complete a thorough assessment to ensure effectiveness and value. In this article, we describe how to evaluate the impact of a health technology innovation through the 4 dimensions of care outlined by the Quadruple Aim Framework, using our experience with the Champlain Building Access to Specialists through eConsultation (BASE) eConsult service as a case example.

Methods: A descriptive overview of data was collected between April 1, 2011, and August 31, 2017, using 4 dimensions of care outlined by the Quadruple Aim Framework: patient experience, provider experience, costs, and population health. Findings were drawn from use data, primary care provider closeout surveys, surveys/interviews with patients and provider, and costing data.

Results: Overall, patients have received access to specialist advice within days and find the advice useful in 86% of cases. Provider experience is very positive, with satisfaction ratings of high/very high value in 94% of cases. The service cost a weighted average of $47.35/case, compared with $133.60/case for traditional referrals. In total, 1,299 primary care providers have enrolled in the service, completing 28,838 cases since 2011. Monthly case volumes have grown from an average of 13 cases/month in 2011 to 969 cases/month in 2016.

Conclusions: The eConsult service has been widely adopted in our region and is currently expanding to new jurisdictions across Canada. However, although we successfully demonstrated eConsult's impact on patient experience, provider satisfaction, and reducing costs, we met several challenges in evaluating its impact on population health. More work is needed to evaluate eConsult's impact on key population health metrics (eg, mortality, morbidity, and system use). Efforts to conduct such evaluations are underway.

Excessive wait times for specialist care are associated with patient and provider frustration, poor health outcomes, and dissatisfaction with care.1⇓⇓–4 To improve access to specialists, many health care organizations have embraced innovative eHealth platforms such as electronic consultation (eConsult) systems: secure web-based tools that offer primary care providers (PCPs) with quick access to specialist advice for their patients.5⇓⇓⇓⇓–10 However, technology solutions are too often implemented without a true understanding of the system-level problem they seek to address, resulting in excessive costs, poor adoption, ineffectiveness, and ultimately failure.11 This was seen in the United Kingdom, where the National Health Service invested in an initiative to provide physicians with a suite of health information technologies, such as comprehensive electronic medical records. Despite a multibillion dollar investment, a recent report from the National Audit Office found that among many of these programs, as much as 98% of their purported benefits have not been achieved.12 Similar failures have been reported in other health systems13, including Canada's, which lags behind other Organization for Economic Co-operation and Development countries in creating coordinated, integrated systems that improve care delivery despite significant investment in health technology over the last decade.14,15 To avoid such missteps, it is important that any region attempting to implement a new technology-based initiative in health care improvement should evaluate its impact across multiple dimensions by using meaningful parameters.

In 2009, our research team implemented the Champlain Building Access to Specialists through eConsultation (BASE) eConsult service, an on-line application that allows PCPs and specialists to communicate asynchronously via a secure channel.16 The service's initial success and potential for expansion led us to adopt a consistent evaluative approach to capture a broad picture of eConsult's impact on care. We selected the Quadruple Aim Framework, a modification of the established Triple Aim Framework of health care improvement17, which describes the importance of pursuing and evaluating 3 equally weighted dimensions of care: improving the health of populations, enhancing patients' experience of care, and reducing costs. Although an effective measure of health system performance at a system level, Triple Aim does not consider provider experience, which has been shown to have a substantial impact on adoption of new technologies, overall quality of care, and retention in remote areas.18 Quadruple Aim addresses this limitation by adding provider experience as a fourth dimension and therefore may offer a more comprehensive approach to the evaluation of health technology.

In this article, we will describe how to evaluate the impact of a health technology innovation through the 4 dimensions of care outlined by the Quadruple Aim Framework (population health, patient experience, provider experience, and costs) by using our experience with the eConsult service16 as a case example. Our goals are to examine the service's impact among the 4 measures outlined by the Quadruple Aim Framework and identify its limitations in evaluating health technology innovations such as eConsult. Our findings will not only be of relevance to health regions and stakeholders implementing eConsult services, but also may help guide the evaluation of other health technology innovations, particularly those that depend on provider adoption for successful implementation.

Methods

Setting

Canada's health care system is publically funded. Canadian citizens are automatically covered for emergent and acute care services through their provincial insurance plans, with the exception of the territories, First Nations groups, and military personnel, which are covered under separate federal plans. Although Canada's health care has been ranked as high quality in some areas, it has faced serious and growing issues with wait times and access to specialist care. A recent study by the Commonwealth Fund found that Canadians faced the longest wait times for specialist care among citizens of the 11 participating countries.19 The United States faces shorter wait times for specialist care, but experiences greater challenges in equality of access, with low-income or uninsured individuals facing significant barriers to prompt care.20

The Champlain BASE eConsult Service

To use the eConsult service, PCPs log onto a secure web-based application by using any device equipped with a web browser. All data are maintained on secure servers using the Microsoft SharePoint platform. The handling of patient information adheres to all requirements outlined by Ontario's Personal Health Information Protection Act.21 PCPs enter their question along with any pertinent case details into the text field, attach any files they deem useful (eg, images and test results), and select a specialty group from a drop-down list. A designated case assigner allocates the case to a specialist, who receives an email notifying them of the case. Specialists are asked to respond within 1 week with advice on how the PCP can manage the case, a recommendation for the patient to be referred, or a request for further information. Discussion can continue between the PCP and specialist until the PCP closes the case. An example of a typical eConsult case is included in Figure 1.

Figure 1.
Figure 1.

Example of a case submitted to a specialist via the Champlain Building Access to Specialists through eConsultation (BASE) eConsult service (details modified to ensure patient anonymity). PCP, principal care provider.

The eConsult service automatically collects usage data whenever a PCP begins a case, identifying such elements as the patient's age and gender, the PCP's practice location and rurality (a measure of population density and travel time to the nearest referral center, calculated on a 0 to 100 scale by using the Rurality Index of Ontario), the specialty referred to, and the specialist's self-reported time spent responding to the case (which is used to calculate specialist remuneration prorated to $200/h). The service also creates a log of all correspondence between the PCP and specialist. Upon closing the case, the PCP completes a brief survey in which they are asked about the case's outcome, whether a referral was originally contemplated and/or ultimately avoided as a result of the specialist's advice, and the eConsult's value to their patients and themselves by using a 5-point Likert scale. PCPs may also leave optional free-text comments. Modifications to the survey made on October 1, 2016, changed questions 3 and 4 to concern the case's educational value and capacity to serve as material for continuing medical education (CME), respectively.

Data Sources

To complete a broad assessment of the eConsult service, we have drawn data from a number of sources, which include the following: (1) use data collected automatically by the service during each exchange, (2) PCPs' responses to a mandatory 5-question closeout survey completed at the end of every case, (3) previously conducted qualitative research using surveys and interviews to assess PCP, specialist, and patient perspectives on eConsult, and (4) costing data, both unique and previously reported. Our results are principally based on all eConsult cases completed between April 1, 2011, (the date at which data collection began) and August 31, 2017, although subsets of these data are used in the case of certain metrics where the full dataset cannot be used. These cases are described clearly where they occur. To address the measures outlined by the Quadruple Aim Framework as fully as possible, we also report on the results of previous studies that use interviews, surveys, cost analyses, and other data sources. Ethics approval for this study was provided by the Ottawa Health Science Network Research Ethics Board and the Bruyère Continuing Care Research Ethics Board.

Results

Table 1 presents an overview of the 4 dimensions of care outlined by the Quadruple Aim, measures and outcomes associated with each dimension, and ways in which eConsult addresses that element of care.

View this table:
Table 1.

The Champlain Building Access to Specialists through eConsultation Service's Impact Viewed Through the Lens of the Quadruple Aim Framework

Patient Experience

During our initial proof-of-concept stage, we conducted surveys with patients attending face-to face specialist appointments in a local endocrinology clinic to assess the wait times they experienced for care.22 Of the 101 patients who completed the survey, 33% waited longer than 6 months for care, 58% experienced significant worry relating to the wait, and 30% reported that their symptoms interfered with their day-to-day lives.22 When asked whether they would accept treatment via an eConsult service, 45% of patients stated that the eConsult service would be an acceptable alternative. Some patients expressed a preference to see a specialist in person, whereas others noted the limitation in the case where the patient did not have a PCP.

To capture patient attitudes and experiences directly once the service was launched, we conducted an in-depth qualitative study with patients treated using eConsult. Thirty patients who had been treated using eConsult completed telephone interviews in which they discussed their experiences with the service and offered their opinions of it in comparison to traditional face-to-face specialist visits. We found that 86% of patients stated eConsult was useful in their situation, 97% considered it an acceptable alternative to face-to-face consultations, and 83% claimed they would ask their PCP to use the service on their behalf in the future.23 Continuity of care emerged as a major theme, with many patients appreciating that the service allowed their PCP to act as an advocate and guide throughout their interactions with the health care system.

The eConsult service also improves patients' experience of care by providing prompt access to specialist advice and avoiding unnecessary referrals. In an evaluation of use data collected over the first 5 years of the eConsult service (April 2011 to April 2016), we found that specialists responded to PCP questions in a median of 0.9 days, 57% of cases resulted in PCPs receiving new or additional information, and 68% of cases were resolved without requiring a face-to-face specialist referral.24 An evaluation of more recent data confirms these findings. Among the 14,460 people who received an eConsult between May 2016 and August 2017, their PCPs received specialist replies in a median of 1.2 days (interquartile range, 1.2 to 4.2), received new or additional information in 58% of cases, and resolved 71% of cases without making a face-to-face referral for the patient.

Provider Experience

The service's impact on provider experience has been captured through surveys, focus groups, interviews, and analysis of PCPs' written comments. At the conclusion of each case, PCPs completed a brief survey assessing its outcome and their opinion on its value, as rated on a 5-point scale (1 being very low and 5 being very high). To date, 94% of PCPs have rated the service as 4 (14%) or 5 (80%) out of 5, signifying high and very high values, respectively. When discussing the service, PCPs frequently cite the speed of responses, quality of advice, capacity for improving patient care, and educational opportunities as its chief benefits.25 Likewise, in a survey completed by 34 specialists from 23 specialty groups who were enrolled with the eConsult service on or before May 20, 2014, the majority of respondents stated that eConsult is a feasible way to improve access to specialist care (94%) and communication (94%), has educational value for PCPs (91%) and is user friendly (82%).26

In our interactions with the PCPs and specialists reported above, providers consistently cited eConsult's capacity to work as an educational tool as one of its chief benefits. To further measure this attitude, 2 questions were added to the PCP closeout survey on October 1, 2016. The first of these asks PCPs to report how helpful and/or educational the specialist's response was in guiding their ongoing evaluation or management of the patient, by using a 5-point Likert scale (with 1 meaning minimal and 5 meaning excellent). For the second question, the survey poses the statement “this eConsult addresses an important clinical problem that should be incorporated into upcoming CME events” and asks users to respond using a 5-point scale (1 meaning strongly disagree and 5 meaning strongly agree). A total of 10,364 cases have been completed since the new questions were added. In 92% of cases, PCPs reported high (n = 2,501) or very high (n = 7,051) educational value. In 57% of cases, PCPs either agreed (n = 3,033) or strongly agreed (n = 2,582) that the case could be incorporated into CME events, while only 8% disagreed (n = 625) or strongly disagreed (n = 157), and 36% were neutral (n = 3,697).

Cost Analysis

Our analysis of eConsult's cost-effectiveness embraced multiple perspectives, including the total cost of the system itself, the comparative cost of an eConsult case versus a traditional referral, and the service's capacity to avoid downstream societal costs to patients and the health care system.

The total system costs over the 2014 to 2015 fiscal year were $207,680, including delivery costs (eg, user set up support and administration) and consultation-specific costs (assignment and specialist remuneration). Across specialty groups, the service cost a weighted average of $47.35/case, compared with $133.60/case for traditional referrals.27 Furthermore, a costing evaluation from the societal perspective calculated an additional savings of $11 per case, when subtracting delivery costs, specialist remuneration, the costs associated with referral initiated as a result of the service from the costs of referrals avoided through eConsult, and indirect patient savings resulted from avoided travel and lost wages/productivity.28 Furthermore, avoidance of other indirect costs, such as avoided tests and improved health outcomes associated with shorter wait times, have not yet been measured, but will be able to be studied as systems mature and will likely account for additional savings.

Population Health

The effects of referrals on traditional measures of population health (eg, mortality, morbidity, and health status) are not well defined within the traditional referral process. Consequently, we were unable to assess eConsult's impact on population health by using these typical measures. For the purpose of this article, we have chosen several alternate metrics pertaining to population health: usage frequency, population served, provider use, patient safety, and equity.

Usage Frequency

Usage of the service has grown. PCPs have completed 28,838 cases since 2011, when the service was implemented as a full pilot and collection of usage data began (Figure 2). Cases are not distributed evenly over the past 5 years, but have gained frequency each year, with 9,604 cases (33% of all cases competed to date) closed in 2016 and 7752 cases closed between January and August of 2017 (27% of all cases). Monthly case volumes have grown from an average of 13 cases/month in 2011 to 969 cases/month in 2017 (Figure 2). The number of available specialty groups has increased from 5 to 105 (Figure 3).

Figure 2.
Figure 2.

Number of eConsult cases completed per month and cumulatively.

Figure 3.
Figure 3.

The number of specialty groups available from the eConsult service by year.

Population Served

The eConsult service is available to PCPs from across Ontario, Canada, and has also been adopted by 19 PCPs in Nunavut, Canada. However, the majority of the service's users are currently located in the Champlain Local Health Integration Network (LHIN), the health region where eConsult is administered and where it began as a regional service. The Champlain LHIN is situated in eastern Ontario, Canada and has a population of 1.3 million people.29 Roughly half of the population resides in a main urban center (Ottawa, Canada), whereas the rest live in surrounding semiurban and rural communities up to 3 hours away by car.

Using data from a recent study recruiting all providers actively delivering primary care services in the region, we estimate that 1,077 family physicians practice in the Champlain LHIN.30 Of this group, 805 physicians have joined the eConsult service, comprising a 75% rate of adoption. Assuming an even distribution of patients across physicians, approximately 975,000 patients in the Champlain LHIN receive care in which eConsult can be used to promptly access specialist advice.

Provider Use

Of the 1299 PCPs (including family physicians and nurse practitioners) who have joined the service, 795 (61%) have completed a case in the past 12 months. These users have completed a median of 8 cases (interquartile range, 3 to 18). The average number of cases completed by PCPs has increased over time, with 1.7 cases/PCP in 2011 versus 10.5 cases/PCP in 2016. In its initial form as a proof-of-concept, the eConsult service provided 18 PCPs with access to advice from 5 specialty groups. Since then, the numbers of users has expanded, with 1299 PCPs (1103 family physicians and 196 nurse practitioners) from 449 clinics in 105 towns/cities enrolled. PCPs were initially selected using a targeted recruitment strategy. However, subsequent enrollment has proceeded largely by word of mouth. Curtailing active recruitment has not decreased the rate of enrollment; indeed, the number of new PCPs enrolled increased steadily from 2011 to 2015 (Figure 4). Enrollment returned to 2014 levels in 2016 and 2017, suggesting that the service is reaching saturation in the Champlain LHIN. Expansion to new jurisdictions across Ontario and other parts of Canada are underway, which will likely increase the frequency of uptake.

Figure 4.
Figure 4.

The number of new principal care provider (PCPs) enrolled in the eConsult service by year.

Patient Safety

The eConsult service has the potential to increase patient safety by alerting PCPs to cases where they had not originally contemplated a referral, but chose to complete one based on the specialist's advice. A recent study found that 3.4% of all eConsult cases reflect this outcome, of which 5% were later deemed urgent.31

Discussion

This article outlines an approach that quality improvement and health system research teams can apply as a guiding framework to their health technology innovation projects. The Quadruple Aim Framework has helped our team maintain a whole-system approach in our ongoing efforts to improve access to specialist advice. Consideration of all 4 aspects of the Quadruple Aim Framework (patient care, provider satisfaction, cost effectiveness, and population health) is important to ensure value. When implementing a health care innovation, especially for technology-based solutions, innovators must ensure that the chosen method will address an articulated problem in a manner that improves patients' experience of care. This is particularly true for services that change the traditional pattern of care delivery, as eConsult does in cases where patients were able to avoid face-to-face specialist visits to which they would have otherwise been referred.

Implementation of any innovation must include a consideration of how patients experience care, both through the innovation itself and during the process the innovation seeks to improve. Our decision to develop the eConsult service was built on a previously articulated need for better access to specialist care.22 Findings from subsequent studies demonstrate that this need extends beyond a single clinic or patient cohort. A survey of 66 patients of a chronic pain clinic in Ottawa revealed that 31% of respondents waited more than a year for care and 94% reported negative effects of wait times on their day-to-day lives.32 More broadly, a series of chart audits conducted in 5 clinics across Ontario found that median wait times were 79 days for nonurgent referrals and 49 days for urgent referrals.33 Once we launched the eConsult service, we sought to capture patients' perspectives on the care it provided to ensure that the service was meeting our goals. Our patient surveys were critical in this regard, as they reassured us of eConsult's overall acceptability while highlighting the roles PCPs and specialists play in the referral-consultation process. This knowledge allowed us to implement the service in a manner that considered and addressed its limitations. For example, patients can only receive care through eConsult if they have a dedicated PCP willing to act on the specialist's advice. Furthermore, patients should have a say in whether or not they are treated via eConsult, as one cannot assume that all patients would prefer an eConsult over a traditional face-to-face referral, which may be seen as a more desirable option in some cases (eg, in the case of patients who are transgender, have HIV, or belong to other marginalized groups).

Although patient perspectives must be considered when implementing a health innovation, provider perspectives are also key. The sense of fulfillment providers get, or fail to get, from their work has a palpable impact on the quality of care they provide.34 When developing health care improvements, many innovators do not consider how the adoption of new tools or processes will affect providers, ignoring such critical factors as workflow and ease of use. In the current health care context, providers are increasingly being asked to provide a greater range of service with fewer resources. Overburdened physicians are more likely to prescribe inappropriate medications, make unnecessary referrals, and diverge from treatment plans, resulting in worse patient outcomes and higher dissatisfaction with care.18 When developing eConsult, we used a clinician-led approach to address a problem that providers themselves had often articulated (ie, long wait times for specialist advice). Providers have expressed an overwhelmingly positive view of eConsult, noting its ability to improve interprovider communication and provide educational support.25 By fostering communication between PCPs and specialists, eConsult services help providers learn from one another and build relationships, factors that have been closely associated with the high levels of satisfaction reported by their users.25,35 Whereas questions asked via traditional referrals often involve minimal interaction between the PCP and specialist36, eConsult allows conversations to emerge between providers, leading to better understanding, more informed PCP questions, and, consequently, more effective specialist responses.26 These connections make eConsult a natural learning tool for PCPs, a fact frequently stated by PCPs and specialists alike.25,26 Our team is currently exploring the service's ability to provide needs assessments for continuing professional development activities.37,38 Its ability to transcend geographic distance is of particular use to rural providers, and eConsult may therefore act as a retention strategy for these regions.

Our evaluation of eConsult's economic impact on care demonstrates a capacity for substantial cost reductions. Health care systems in many nations face a financial crunch and must cut back on key services to remain within budget. Innovators must therefore consider all costs when developing health care improvements, including infrastructure, information technologies and administrative support, and adoption efforts.

When applying the Quadruple Aim Framework to our evaluation of the eConsult service, we were unable to use the traditional metrics of population health, which include mortality, morbidity, and emergency department use. This was the largest challenge of our analysis and the chief limitation of our study. To assess eConsult's impact on traditional population health metrics, a sufficiently large portion of a given population must receive treatment via the service so that their outcomes may be measured against those who received usual care (ie, a control group). As eConsult is a relatively new innovation, most services have not yet generated sufficient case volume to accurately assess such outcomes39,40, although preliminary results are encouraging.41,42 For example, a randomized controlled trial conducted in a community health center in Connecticut found that treatment via a cardiology eConsult service was associated with fewer cardiac-related visits to the emergency department.42 In addition, our team is exploring eConsult's impact on adherence to best-practice guidelines in select patient populations with chronic pain and kidney disease. However, further exploration of this area is warranted.

As we were unable to report on patient health outcomes directly, we endeavored to explore eConsult's potential impact on population health through measures of its ability to reach patients (eg, population served and provider use) and other beneficial effects on the target population (eg, patient safety). In measures of its reach, we estimated that 975,000 patients in the Champlain LHIN are enrolled with a PCP who uses eConsult, and that 61% of enrolled PCPs are active users (ie, have submitted a case in the past 12 months). These figures align with other services reported in the literature: the Kaiser Permanente Network serves half a million patients in its Colorado location6 and over 3 million in California43, and a teledermatology service in the Netherlands identified 65% of PCPs as active users (ie, those who had completed at least one case) who submitted an average of 9.1 cases per year.44 We also found that in a small percentage of cases (3.4%), PCPs did not originally intend to refer the patient, but ultimately chose to do so based on the advice they received. This has important ramifications for patient safety, as delayed referrals can often have significant consequences for patients.45,46

The eConsult service may also help to reduce inequity of access to specialist advice for individuals from rural or remote areas where specialists are less likely to practice, as well as individuals facing socioeconomic barriers to accessing care (eg, homelessness, addiction issues, and lack of fluency in English or French).47 By allowing these individuals to potentially receive specialist advice through their PCPs, eConsult can reduce the burden of seeking care from a provider with whom they are unfamiliar, and who in some cases may practice in another city hundreds of kilometers away.48

Our study has several limitations in addition to its inability to capture health outcomes as described above. Reports of patient and provider satisfaction may have been subject to selection bias, as individuals who support the service may be more inclined to use it and to respond to interviews and surveys. This limitation is somewhat mitigated in the closeout surveys, which are mandatory and thus guarantee full participation; however, we recognize the limitations of mandatory surveys, which may not always reflect the exact views of the responders. As such, our findings should be interpreted with caution.

Conclusion

Using the Quadruple Aim Framework, we demonstrated eConsult's impact on patient experience, provider satisfaction, and lowered cost. Assessing its impact on population health remains a challenge, and although proxy measures show encouraging signs, more data are needed to assess its effects on mortality and morbidity. Applying the Quadruple Aim Framework to guide the evaluation of health technology innovations, such as eConsult, provides a comprehensive, in-depth approach to support the pursuit of value for health care and health system optimization. Although further evidence is needed to assess eConsult's impact on clinical population outcomes, current research is sufficient to demonstrate eConsult's potential as a transformative innovation.

Acknowledgments

We thank the health care providers who participated in eConsult, the Winchester District Memorial Hospital for hosting the service and providing technical support, and Justin Joschko for his contributions in drafting and editing the manuscript.

Notes

  • This article was externally peer reviewed.

  • Funding: Funding for this project was provided through the Ontario Ministry of Health and Long-Term Care, and the INSPIRE-PHC Applied Health Research Question. The funders had no role in study design, data collection/analysis/interpretation, or preparation of the manuscript.

  • Conflict of interest: none declared.

  • To see this article online, please go to: http://jabfm.org/content/31/3/445.full.

  • Received for publication October 6, 2017.
  • Revision received January 11, 2018.
  • Accepted for publication January 21, 2018.

References

  1. 1.↵
    Canadian Institute for Health Information. Health care in Canada, 2012: A focus on wait times. Ottawa, ON: Canadian Institute for Health Information; 2012. Available from: https://secure.cihi.ca/free_products/HCIC2012-FullReport-ENweb.pdf. Accessed March 2, 2017.
  2. 2.↵
    1. Barua B,
    2. Esmail N
    . Waiting your turn: Wait times for health care in Canada. Vancouver, BC: Fraser Institute; 2013. Available from: https://www.fraser institute.org/sites/default/files/waiting-your-turn-2013.pdf. Accessed March 2, 2017.
  3. 3.↵
    1. Chen AH,
    2. Yee HF
    . Improving the primary care to specialty care interface: Getting from here to there. Arch Intern Med 2009;169:1024–6.
    OpenUrlCrossRefPubMed
  4. 4.↵
    1. Barua B,
    2. Esmail N,
    3. Jackson T
    . The effect of wait times on mortality in Canada. Vancouver, BC: Fraser Institute; 2014. Available from: https://www.fraserinstitute.org/sites/default/files/effect-of-wait-times-on-mortality-in-canada.pdf. Accessed March 2, 2017.
  5. 5.↵
    1. Chen AH,
    2. Murphy EJ,
    3. Yee HF Jr.
    . eReferral–a new model for integrated care. N Engl J Med 2013;368:2450–3.
    OpenUrlCrossRefPubMed
  6. 6.↵
    1. Palen TE,
    2. Price D,
    3. Shetterly S,
    4. et al
    . Comparing virtual consults to traditional consults using an electronic health record: An observational case-control study. BMC Med Inform Decis Mak 2012;12:65.
    OpenUrlCrossRefPubMed
  7. 7.↵
    1. Kim Y,
    2. Chen AH,
    3. Keith E,
    4. et al
    . Not perfect, but better: Primary care providers' experiences with electronic referrals in a safety net health system. J Gen Intern Med 2009;24:614–9.
    OpenUrlCrossRefPubMed
  8. 8.↵
    1. Kim-Hwang JE,
    2. Chen AH,
    3. Bell DS,
    4. et al
    . Evaluating electronic referrals for specialty care at a public hospital. J Gen Intern Med 2010;25:1123–8.
    OpenUrlCrossRefPubMed
  9. 9.↵
    1. McAdams M,
    2. Cannavo L,
    3. Orlander JD
    . A medical specialty e-consult program in a VA health care system. Fed Pract 2014;31:26–31.
    OpenUrl
  10. 10.↵
    1. Rodriguez KL,
    2. Burkitt KH,
    3. Bayliss NK,
    4. et al
    . Veteran, primary care provider, and specialist satisfaction with electronic consultation. JMIR Med Inform 2015;3(1):e5.
    OpenUrlCrossRef
  11. 11.↵
    1. Ammenwerth E,
    2. Talmon J,
    3. Ash JS,
    4. et al
    . Impact of CPOE on mortality cates—Contradictory findings, important messages. Methods Inf Med 2006;45:586–94.
    OpenUrlPubMed
  12. 12.↵
    National Audit Office. Review of the final benefits statement for programs previously managed under the National Programme for IT in the NHS. London, United Kingdom: NAO; 2013. Available from: https://www.nao.org.uk/wp-content/uploads/2013/06/10171–001_NPfiT_Review.pdf. Accessed March 2, 2017.
  13. 13.↵
    1. Johnson CW
    . Case studies in the failure of healthcare information systems. Glasgow, United Kingdom: University of Glasgow; 2010. Available from: http://www.dcs.gla.ac.uk/∼johnson/papers/AHRQ/case_study.pdf. Accessed March 2, 2017.
  14. 14.↵
    1. Liddy C,
    2. Hogel M,
    3. Blazhko V,
    4. et al
    . The current state of electronic consultation & electronic referral systems in Canada: An environmental scan. Stud Health Technol Inform 2015;209:75–83.
    OpenUrl
  15. 15.↵
    1. Chang F,
    2. Gupta N
    . Progress in electronic medical record adoption in Canada. Canadian Family Physician 2015 Dec 1;61:1076–84.
  16. 16.↵
    1. Champlain BASE
    . Champlain BASE eConsult service. Available from: http://champlainbaseeconsult.com/ Accessed April 28, 2017.
  17. 17.↵
    1. Berwick D,
    2. Nolan T,
    3. Whittington J
    . The triple aim: Care, health and cost. Health Aff 2008;27:759–69.
    OpenUrlAbstract/FREE Full Text
  18. 18.↵
    1. Bodenheimer T,
    2. Sinsky C
    . From Triple to Quadruple Aim: Care of the patient requires care of the provider. Ann Fam Med 2014;12:573–6.
    OpenUrlAbstract/FREE Full Text
  19. 19.↵
    Canadian Institute for Health Information. How Canada compares: Results from the Commonwealth Fund's 2016 International Health Policy Survey of adults in 11 countries. Available from: https://www.cihi.ca/sites/default/files/document/text-alternative-version-2016-cmwf-en-web.pdf. Accessed April 28, 2017.
  20. 20.↵
    1. Davis K,
    2. Ballreich J
    . Equitable access to care–how the United States ranks internationally. N Engl J Med 2014;371:1567–70.
    OpenUrlCrossRefPubMed
  21. 21.↵
    Government of Ontario. Personal Health Information Protection Act. Available: https://www.ontario.ca/laws/statute/04p03. Accessed July 21, 2017.
  22. 22.↵
    1. Keely E,
    2. Traczyk L,
    3. Liddy C
    . Patient perspectives on wait times and the referral-consultation process while attending a tertiary diabetes and endocrinology centre: Is eConsultation an acceptable option? Can J Diabetes 2015;39:325–9.
    OpenUrl
  23. 23.↵
    1. Joschko J,
    2. Liddy C,
    3. Moroz I,
    4. et al
    . Just a click away: Exploring patients' perspectives on receiving care through the Champlain BASETM eConsult service. Fam Pract 2017;35:93–98.
    OpenUrl
  24. 24.↵
    1. Liddy C,
    2. Moroz I,
    3. Afkham A,
    4. Keely E
    . Sustainability of a primary care driven eConsult service. Ann Fam Med 2018;16:120–28.
    OpenUrlAbstract/FREE Full Text
  25. 25.↵
    1. Liddy C,
    2. Afkham A,
    3. Drosinis P,
    4. et al
    . Impact and satisfaction with a new eConsult service: A mixed methods study of primary care providers. J Am Board Fam Med 2015;28:394–403.
    OpenUrlAbstract/FREE Full Text
  26. 26.↵
    1. Keely E,
    2. Drosinis P,
    3. Afkham A,
    4. et al
    . Perspectives of Champlain BASE specialist physicians: Their motivation, experiences and recommendations for providing eConsultations to primary care providers. Stud Health Technol Inform 2015;209:38–45.
    OpenUrlPubMed
  27. 27.↵
    Champlain BASE. Fact Sheets. Available from:http://www.champlainbaseeconsult.com/fact-sheets. Accessed July 31, 2017.
  28. 28.↵
    1. Liddy C,
    2. Drosinis P,
    3. Deri Armstrong C,
    4. et al
    . What are the cost savings associated with providing access to specialist care through the Champlain BASE eConsult service? A costing evaluation. BMJ Open 2016;6(6):e010920.
    OpenUrlAbstract/FREE Full Text
  29. 29.↵
    1. Champlain LHIN
    . Sub-Region Population Health Profiles. Available from: http://www.champlainlhin.on.ca/∼/media/sites/champlain/Goals_Achvmnts/IHSP/SubRegions/Data/201712SRHlthPrflsTechRptEN.pdf?la=en. Accessed April 4, 2018.
  30. 30.↵
    1. Johnston S,
    2. Liddy C,
    3. Hogg W,
    4. et al
    . Barriers and facilitators to recruitment of physicians and practices for primary care health services research at one centre. BMC Med Res Methodol 2010;10:109.
    OpenUrlCrossRefPubMed
  31. 31.↵
    1. Liddy C,
    2. Drosinis P,
    3. Fogel A,
    4. et al
    . Prevention of delayed referrals through the Champlain BASE eConsult Service. Can Fam Physician 2017;63(8):e381–e386.
    OpenUrlAbstract/FREE Full Text
  32. 32.↵
    1. Liddy C,
    2. Poulin PA,
    3. Hunter Z,
    4. Smyth C,
    5. Keely E
    . Patient perspectives on wait times and the impact on their life: A waiting room survey in a chronic pain clinic. Scand J Pain 2017;17:53–7.
    OpenUrl
  33. 33.↵
    1. Liddy C,
    2. Nawar N,
    3. Moroz I,
    4. et al
    . Understanding patient referral wait times for specialty care in Ontario: A retrospective chart audit. Healthcare Policy 2018;13:59–69.
    OpenUrl
  34. 34.↵
    1. Wallace JE,
    2. Lemaire JB,
    3. Ghali WA
    . Physician wellness: a missing quality indicator. The Lancet. 2009;374:1714–21.
    OpenUrl
  35. 35.↵
    1. Rodriguez KL,
    2. Burkitt KH,
    3. Bayliss NK,
    4. et al
    . Veteran, primary care provider, and specialist satisfaction with electronic consultation. JMIR Med Inform 2015;3(1):e5.
    OpenUrlCrossRef
  36. 36.↵
    1. Conley J,
    2. Jordan M,
    3. Ghali WA
    . Audit of the consultation process on general internal medicine services. Qual Saf Health Care 2009;18:59–62.
    OpenUrlAbstract/FREE Full Text
  37. 37.↵
    1. Tran C,
    2. Liddy C,
    3. Pinto N,
    4. et al
    . Impact of question content on e-consultation outcomes. Telemed J E Health 2016;22:216–22.
    OpenUrl
  38. 38.↵
    1. Tran C,
    2. Liu DM,
    3. Afkham A,
    4. et al
    . Harnessing e-consultations to improve practice-based learning in endocrinology. Can J Diabetes 2015;6:539.
    OpenUrl
  39. 39.↵
    1. Liddy C,
    2. Drosinis P,
    3. Keely E
    . Electronic consultation systems: Worldwide prevalence and their impact on patient care–a systematic review. Fam Pract 2016;33:274–85.
    OpenUrlCrossRefPubMed
  40. 40.↵
    1. Vimalananda VG,
    2. Gupte G,
    3. Seraj SM,
    4. et al
    . Electronic consultations (e-consults) to improve access to specialty care: A systematic review and narrative synthesis. J Telemed Telecare 2015;21:323–30.
    OpenUrlCrossRefPubMed
  41. 41.↵
    1. Wrenn K,
    2. Catschegn S,
    3. Cruz M,
    4. et al
    . Analysis of an electronic consultation program at an academic medical centre: Primary care provider questions, specialist responses, and primary care provider actions. J Telemed Telecare 2016;23:217–224.
    OpenUrl
  42. 42.↵
    1. Olayiwola JN,
    2. Anderson D,
    3. Jepeal N,
    4. et al
    . Electronic consultations to improve the primary care-specialty care interface for cardiology in the medically underserved: a cluster-randomized controlled trial. Ann Fam Med 2016;14:133–40.
    OpenUrlAbstract/FREE Full Text
  43. 43.↵
    1. Kahn E,
    2. Sossong S,
    3. Goh A,
    4. et al
    . Evaluation of skin cancer in northern California Kaiser Permanente's store-and-forward teledermatology referral program. Telemed J E Health 2013;19:780–5.
    OpenUrlCrossRefPubMed
  44. 44.↵
    1. van der Heijden JP,
    2. de Keizer NF,
    3. et al
    . Teledermatology applied following patient selection by general practitioners in daily practice improves efficiency and quality of care at lower cost. Br J Dermatol 2011;165:1058–65.
    OpenUrlCrossRefPubMed
  45. 45.↵
    1. Kim DH,
    2. Kim M,
    3. Kim H,
    4. et al
    . Early referral to a nephrologist improved patient survival: Prospective cohort study for end-stage renal disease in Korea. PLoS One 2013;8(1):e55323.
    OpenUrlCrossRefPubMed
  46. 46.↵
    1. van der Linden MP,
    2. Le Cessie S,
    3. Raza K,
    4. et al
    . Long-term impact of delay in assessment of patients with early arthritis. Arthrit Rheum 2010;62:3537–46.
    OpenUrlCrossRefPubMed
  47. 47.↵
    1. Karunanayake CP,
    2. Rennie DC,
    3. Hagel L,
    4. et al
    . Access to specialist care in rural Saskatchewan: The Saskatchewan rural health study. Healthcare (Basel) 2015;3:84–99.
    OpenUrl
  48. 48.↵
    1. Sabesan S
    . Specialist cancer care through telehealth models. Aust J Rural Health 2015;23:19–23.
    OpenUrl
Back to top

In this issue

Print
Download PDF
Article Alerts
Email Article
Citation Tools

Jump to section

Keywords