Abstract
Background: Pragmatic clinical trials (PCTs) are increasingly recommended to evaluate interventions in real-world conditions. Although PCTs share a common approach of evaluating variables from actual clinical practice, multiple characteristics can differ. These differences affect interpretation of the trial. The Pragmatic-Explanatory Continuum Indicator Summary (PRECIS) model was developed in 2009 by the CONSORT Work Group on Pragmatic Trials, published by Thorpe et al, to aid in trial design. PRECIS provides clarity about the generalizability and applicability of a trial by depicting multiple study characteristics. We recently completed a National Institutes of Health–sponsored pilot study examining health-related outcomes for 2 complementary therapies for chronic low back pain in patients referred by primary care providers in the Kentucky Ambulatory Network. In preparation for a larger study, we sought to characterize the pragmatic features of the study to aid in our design decisions. The purpose of this article is to introduce clinical researchers to the PRECIS model while demonstrating its application to refine a practice based research network study.
Method: We designed an exercise using an audience response system integrated with a Works in Progress presentation to experienced researchers at the University of Kentucky to examine our study methodologies of parameters suggested by the PRECIS model.
Results: The exercise went smoothly and participants remained engaged throughout. The study received an overall summary score of 30.17 (scale of 0 to 48; a higher score indicates a more pragmatic approach), with component scores that differentiate design components of the study. A polar chart is presented to depict the pragmatism of the overall study methodology across each of these components.
Conclusions: The study was not as pragmatic as expected. The exercise results seem to be useful in identifying necessary refinements to the study methodology that may benefit future study design and increase generalizability. Readers can identify how the PRECIS model may be used to provide clarity and transparency for proposed or existing studies and may wish to replicate our exercise in planning their own studies.
- Chronic Disease
- Practice-based Research
- Pragmatic Clinical Trials
- Quality of Life
- Quantitative Evaluation
Most interventions are evaluated in tightly controlled conditions, creating a gap between medical research and practice that has been recognized as a major factor in the US health care “quality chasm.”1,2 Pragmatic clinical trials (PCTs) are specifically designed to evaluate interventions in real-world conditions. They are considered a solution to the gap between research and practice. However, merely labeling a study as pragmatic is not sufficient; while some study factors (eg, the setting in which the study is conducted) may suggest that it is pragmatic, multiple study characteristics can affect the interpretation of results and their applicability to practice.
One tool that may give a richer, closer look at a trial's characteristics is the Pragmatic-Explanatory Continuum Indicator Summary (PRECIS) model devised by Thorpe et al.3 Criteria for the PRECIS model was developed by the CONSORT Work Group on Pragmatic Trials3,4 to provide clarity about how generalizable and clinically applicable a trial is (pragmatic) versus the extent to which the trial is more focused on basic mechanisms (explanatory).
The PRECIS model typically consists of 10 component dimensions: flexibility of comparison condition; flexibility of experimental intervention; practitioner expertise (in both experimental and comparison conditions); eligibility criteria; primary analysis; practitioner adherence; participant compliance; follow-up intensity; and outcomes. Application of the model involves the formation of scores for each dimension, resulting in dimension and total scores representing the extent to which the dimension is pragmatic. Results of PRECIS scoring often are depicted on a wheel chart, where spokes represent each dimension. PRECIS component scoring that reveals more pragmatic study aspects have plot points that extend toward the outer rim of the wheel's spoke, whereas components that score as more explanatory plot nearer to the center. This creates a useful image that readily allows visual comparison of studies. Figure 1 shows examples of pragmatic and explanatory PRECIS wheels.
Examples of pragmatic-explanatory continuum indicator summary (PRECIS) wheels. A: A PRECIS wheel that is more pragmatic in approach, as evident by the large, round wheel with spoke points near or at the outer rim (4 on the continuum). B: A PRECIS wheel that is more explanatory in approach, as evident by the tight, small, round wheel with spoke points near or at the inner hub (0 on the continuum).
The PRECIS developers intended the model to provide comprehensive information to evaluate the applicability of study results to a particular clinical context and to guide intervention and trial design. Accordingly, the purpose of this article is to provide clinical researchers with an introduction to the PRECIS model while demonstrating its application in refining a practice-based research network study. The use of PCTs is likely to increase with recent initiatives from the Institute of Medicine,5 the US Patient-Centered Outcomes Research Institute,6 and the Agency for Health care Research and Quality,7 expanding support for PCTs and recommending their use and application. With this emphasis on pragmatic approaches, it seems likely that the research community will expect more transparency related to pragmatic characteristics, conditions, and assessments when reporting or proposing studies.4,7,8
Demonstrating PRECIS Application: Methodology
We recently completed a National Institutes of Health–sponsored observational trial that used the Kentucky Ambulatory Network, an established practice-based research network, to refer patients with chronic low back pain (CLBP) to a course of either massage therapy (MT) provided by a community massage practitioner (CMP) or progressive muscle relaxation (PMR). The Kentucky Pain and Research Outcomes Study (KYPROS) was a pilot and feasibility trial that used a 2-arm, repeated measures design, as illustrated in Figure 2. We are planning our next phase of research that will expand the study's size beyond our region to permit a comparison group and generalizability to other populations. To plan our study, we engaged experienced faculty in an exercise with the PRECIS model to compare and contrast study methodologies with those of randomized controlled trials. This would allow us to further refine different aspects of the study to better reach our research goals.
Kentucky Pain and Research Outcomes Study (KYPROS) Design.
Background on Method
The literature on application of the PRECIS model was reviewed to determine methods for our exercise. Three articles were noted; they all found that the PRECIS model provides a richer understanding of the study of interest but differed in their use of the PRECIS tool.
Koppenaal and colleagues9 sought to evaluate the suitability of lifestyle interventions for delivery in clinical practices. Their procedure involved 2 reviewers who independently rated each study to form PRECIS scale scores. Scores then were compared, and consensus was determined. A third researcher was consulted if agreement was not obtained.
Glasgow and colleagues10 used 9 reviewers to score independently 3 weight-loss trials. Interestingly, 7 of the reviewers were members of the investigative teams for respective trials. Most reviewers had terminal degrees and at least moderate research experience. They first read a seminal article on the PRECIS model by Thorpe et al3 then reviewed a slide show created by Sackett.11 They then reviewed protocol materials and background descriptions on each intervention and, if desired, contacted personnel working on the studies for further information. Finally, they independently formed PRECIS scale scores for each intervention, which were combined into group average scores. Results demonstrated the ability of the PRECIS model to differentiate the studies and revealed that there was bias among the respondents, who tended to rate the studies they were personally involved with as more pragmatic.
Riddle and colleagues12 applied PRECIS for a purpose similar to ours, using it as a tool to organize discussion to refine an existing research design. In this case, the group comprised team members already involved in the project; the project had just been funded, and refinement with the PRECIS was part of their plan. Their procedure involved review of the article by Thorpe et al3 and discussion of the PRECIS criteria. Group members then identified their ideal design for the study on a PRECIS wheel. Scores were averaged and then presented to the group, which used this information to consider design changes. They noted that their process resulted in a less pragmatic design, as judged by the final version of the PRECIS wheel, and that the exercise facilitated discussion of and consensus on the project design.
Adaptation of the PRECIS Model
Initial conceptualization of the PRECIS model acknowledged that wheel spokes can be modified to fit specific research designs.3 Indeed, previous researchers adapted the PRECIS to apply it to systematic reviews9 and retrospectively to determine the pragmatism of an ongoing approach.13 To date, the PRECIS tool has been applied mainly to randomized trials that contain an experimental intervention and a comparison intervention. However, in line with our funding source, our study was not randomized nor a clinical trial and was atypical because of its examination of 2 different interventions without the intent of comparing them. To account for this, we modified the PRECIS model to comprise a wheel with 12 spokes (as opposed to 10), with terminology from the component titles described in previous work3 adapted to accommodate the KYPROS design. Specifically, the “flexibility of the experimental/comparison intervention” and “practitioner expertise (experimental/comparison)” components for our PRECIS substituted “MT” and “PMR” for “comparison” and “experimental,” respectively. Furthermore, because MT and PMR participants and providers had several distinctive characteristics, they were not combined for scoring on the continuum for the compliance and adherence components. Instead, individual spokes account for MT and PMR participant compliance and MT and PMR practitioner adherence. We implemented a final modification regarding component intent for the expertise of PMR practitioners and the flexibility of the PMR intervention.
Application Procedure
To apply the PRECIS model, we collected the opinions of experienced researchers during a works in progress presentation in the Department of Family and Community Medicine at the University of Kentucky. Because we were collecting opinions about a research study rather than about the people involved in the research, the University of Kentucky Office of Research Integrity determined that the activity was not human subjects research.
The stated objectives of the presentation were (1) to inform the audience of the research project, (2) to discuss pragmatic versus explanatory research concepts, 3) to apply said concepts through use of the PRECIS model, and (4) to obtain PRECIS scores for the study via the collection of scores from a subset of the group who rated themselves as experienced researchers. The objectives of the presentation were obtained using a PowerPoint presentation we created (available upon request). The PowerPoint included multiple slides taken directly or modified from the slideshow by Sackett11 and used by Glasgow et al10 (described earlier). The presentation included an interactive portion using an audience response system (TurningPoint, version 5.2.1.3179, released July 1, 2013; Turning Technologies, LLC, Youngstown, OH). Fourteen attendees accepted audience response clickers and provided their rating of where on the explanatory (0) to pragmatic (4) scale our study fell for 11 of our 12 PRECIS spokes. The spoke excluded during attendee response was that for PMR practitioner adherence. In the study, PMR was delivered in the form of the recorded voice of a highly experienced PMR practitioner, administered via personal digital assistant. We used this spoke as an example during the presentation; it was selected because it was an obvious zero (explanatory) on the scale because of its consistent delivery, which is inherent in a recorded script.
Before presenting the PRECIS spokes, audience members self-rated their research experience on a 1 to 5 scale, with 1 being no experience and 5 being highly experienced with research activities central to job or leadership of National Institutes of Health–funded research. While all audience members were eligible to score the spokes using their clickers, only the scores from those ratings themselves as a 4 or 5 on the research experience scale are incorporated in results. To maintain the independence of participants, audience responses were not revealed until all spokes had been scored. As each spoke was rated, dialog regarding how and why relevant aspects of the study fit on each spoke occurred. Aspects of this dialog further informed the interpretations of the PRECIS exercise results and are described in the Discussion, below.
Results
All members participating in the audience response (n = 14) remained until the end of the 1-hour presentation. Self-ratings of research experience ranged from 1 (low) to the maximum of 5 (high). Only PRECIS values from the 6 participants who self-rated their research experience at a level of 4 or 5 were used to calculate the overall PRECIS scores for KYPROS. Table 1 provides the mean and median scores for each PRECIS spoke (with standard deviations) from our respondents, a description of each component, and aspects of relevant KYPROS methodology. Figure 3 is a polar chart forming the PRECIS wheel for KYPROS, as indicated by the mean of the respondents' scores for each component.
Pragmatic-explanatory continuum indicator summary (PRECIS) wheel for the Kentucky Pain and Research Outcomes Study (KYPROS). This polar chart indicates the mean score for 6 respondents on each of KYPROS's PRECIS component spokes. Points closer to the outer edge of the wheel indicate more pragmatic attributes (higher scores), whereas those nearer the center circle indicate more explanatory attributes (lower scores) along the continuum. MT, massage therapy; PMR, progressive muscle relaxation.
Our 6 respondents and 12 PRECIS spokes resulted in a possible summary score ranging from 0 (the approach of all elements of the study are as explanatory as possible) to 48 (the approach of all elements of the study are as pragmatic as possible). The sum of the mean scores for each KYPROS PRECIS spoke was 30.17, which indicates that according to our respondents, our study achieved approximately 63% of its potential “pragmatic-ness.”
Discussion
Interpretation of Scores
The summary score provides an overall indicator of study “pragmatic-ness.” High summary scores indicate that the study had few restrictions and thus is more pragmatic. Low summary scores reveal many restrictions in the study and that it is more explanatory. While a summary score may be useful when comparing studies, it may hide spoke scores with disparate values and may be most useful for comparing similar types of studies, such as those that are alike enough to be grouped together in a systematic review.
The PRECIS wheel provides additional information. According to the developers, large wheels suggest the study is pragmatic and that results apply to a large proportion of the target population (Figure 1A). Small PRECIS wheels suggest studies are more explanatory (Figure 1B) and that results may apply to only a small proportion of the target population. An irregular or inconsistently shaped wheel such as ours assists in identifying areas of weaknesses in the protocol.3
Refinement of the Study Based on Results
The KYPROS PRECIS wheel and individual spoke scores surprised us. As suggested by others,10 our familiarity with the study, in addition to the desire to be more pragmatic, may have biased us in thinking our overall study design was more pragmatic than it was. This exercise was particularly helpful in highlighting the areas in which the KYPROS design can improve in its pragmatic-ness. We had expected some irregularity in the wheel, given the major differences in delivery of the intervention between the MT and PMR arms. Otherwise, we expected a large and more uniform PRECIS, suggesting a highly pragmatic study. Instead, our respondents rated several spokes as more explanatory than we expected. Discerning these differences will inform several refinements in the design of our future work.
Two aspects we had thought highly pragmatic were participant characteristics and the study characteristics related to the MT arm. Regarding participant characteristics, we believed that our criteria permitting referred patients with CLBP to have multiple comorbidities and to continue medications and the treatment plan determined by their primary care physician (PCP) reflected real-world patients with CLBP. Our respondents indicated their agreement that our inclusion and exclusion criteria were open; however, their relatively low scores for this dimension (mean, 2.17; standard deviation, 0.69; median, 2; range, 1–3) lead us to reflect more on the methods by which PCPs referred patients into the study. We realized that this aspect of the study was more restrictive than we thought because patients with CLBP were referred by their PCPs only if the patient happened to have an appointment during the referral window at the PCP's site (each PCP site had a 6-week window of time in which they could refer patients to the study). Furthermore, PCPs had to remember the patient had CLBP (the appointment did not have to be related to CLBP), make the recommendation, complete a pocket card, and determine whether the patient would like to be contacted by study personnel. While the exclusion/inclusion criteria for the study reflected “openness,” we speculate that our experienced responders thought that the referral process involving the PCP created a significant barrier to study entry, with possible impact on patient heterogeneity. Although our study had a high number of referring PCPs (n = 41) from 18 study sites, respondents may have felt our reach, particularly in rural areas (6 PCPs from 4 study sites), was not as robust as would be needed for a “real-world” patient sample of Kentucky, which is primarily a rural state. These are areas that we will seek to improve in our future study designs to ensure that results can inform clinical practice.
With regard to the MT arm of the study, responders rated MT intervention flexibility as very pragmatic. Responses differed for MT practitioner expertise and MT participant compliance. Regarding MT practitioner expertise, study therapists were required to have at least 5 years of massage experience. In light of this, we speculate that respondents may have considered these requirements too restrictive for this aspect on the pragmatic side of the continuum, even though no special training except the requirement of holding a state license was needed. Furthermore, our respondents may have believed that personal invitations from study personnel (MT liaisons who were themselves CMPs) to CMPs to participate may have been tantamount to targeted selection for participation, potentially biasing this aspect. While we considered it important that our massage liaisons had an “in” into the massage community,14 we acknowledge that greater effort to limit this potential bias is needed in our future study designs. In addition, detailed consideration of responses to this component also informs us that we may want to consider a less restrictive minimum of CMP experience. However, a more explanatory characteristic within an otherwise pragmatic trial may have an important purpose and/or good rationale, depending on the research question, and should be considered in the development of a study design. Regardless, the shape of a PRECIS wheel can quickly point a researcher or clinician to this aspect for applicable interpretation.
We acknowledge the limitations of our speculative considerations of the respondent scores described above. Specifically, we formed some of our conclusions based on our interpretation of the respondent's scores without confirming qualitative data. However, the value of the PRECIS wheel exercise is demonstrable because we would not have considered these issues without it.
Conclusion
This article is meant to serve multiple purposes. First, we intend to introduce the PRECIS model to practitioner researchers to whom it is unfamiliar. In addition, we seek to explain the PRECIS model and provide examples and references of its versatile and useful applicability. Finally, we demonstrate a practical and replicable application of the PRECIS model through the reporting of an exercise we conducted to assist in the development of future work. We believe each of these objectives are useful, especially for others looking for a similar means to easily and systematically examine their study approach and design.
To date, the PRECIS tool has been considered mainly for typical trials that contain an experimental intervention and a comparison intervention. Our study design was not typical in this regard because of our funding source and study purpose. However, we embraced and used the PRECIS tool as a way to determine the extent to which our design met pragmatic specifications, especially to inform the next steps of our research. Because of the differences between our study and those reviewed in the literature (ie, not having an experimental and comparison intervention group), we altered some spoke title terminology and added spokes to accommodate the additional aspects of research included in our study. These additions enhance our ability to refine our next steps. Our approach to collecting opinions (scores) on PRECIS spokes should be easy for others to replicate. In our next research proposal, we will discuss these scores and the concomitant modifications we plan. Thus, our results should not only enhance the methodology we plan but also increase the clarity and transparency of our design. We recommend the PRECIS wheel exercise for a full a priori or post hoc examination of a study's characteristics.
Acknowledgments
The authors thank Kevin Pearce, MD, MPH; Margaret Love, PhD; Geza Bruckner, PhD; David Greene, MD; Katie Stewart, LMT; and Maureen Flannery, MD, for their contributions to this work; all served advisory or other roles in the KYPROS study. The authors also thank Karen L. Roper, PhD, who provided technical support for the audience response system and reviewed the final manuscript submission.
Notes
This article was externally peer reviewed.
Funding: Support for this study was provided by the National Center for Complementary and Alternative Medicine (grant no. R21AT004544), the National Center for Advancing Translational Sciences, and the National Institutes of Health (grant no. UL1 TR000117).
Conflict of interest: none declared.
- Received for publication January 22, 2014.
- Revision received June 6, 2014.
- Accepted for publication June 23, 2014.
In this issue
Jump to section
Related Articles
Cited By...
- No citing articles found.