Strategies for Planning and Launching PBRN Research Studies: A Project of the Academy of Family Physicians National Research Network (AAFP NRN)

Selecting Fundable, Feasible Studies

Because of the large number of projects that may be under consideration at any one time in the AAFP NRN, a multistep process for “vetting” studies has been developed. In the true spirit of PBRNs, this process incorporates input from both AAFP NRN staff and network members. A variety of issues influence study selection. A proposed study must be a topic of interest to members, of importance to the discipline, and feasible for the NRN. Figure 1 outlines the general process for evaluating study ideas and launching new studies, although this process may vary for a particular study.

• Download figure
• Open in new tab

Figure 1.

The planning and study selection process for studies conducted in the AAFP NRN.

Study ideas in the AAFP NRN come from a variety of sources, including directly from clinician members (the traditional bottom-up concept), principal investigators external to the AAFP NRN, potential funders, AAFP NRN staff, other PBRNs, and indirectly from requests for applications/proposals. Each study concept, regardless of the source of the idea, is first evaluated by the AAFP NRN′s senior leaders (network director, research director, associate research director, and senior scientists). They review available documents including protocols, outlines, and abstracts, and address 4 main questions: (1) Is this a researchable study question? (2) Is the AAFP NRN the best place to answer the question? (3) Does this study fit with the goals and objectives of the NRN? and (4) Is this study fundable?/Do we have the available resources?

If the senior leadership agrees that the idea meets these 4 criteria, a 1-page concept overview is presented to 1 of the 2 AAFP NRN Scientific Review Committees (SRC). Each SRC consists of 6 AAFP NRN physicians and 1 study coordinator (usually a nurse or medical assistant). Proposals are distributed to the 2 SRCs on a rotating basis. The SRC’s objective is to determine whether the study proposal is suitable and feasible in a primary care practice. SRC members may also make suggestions on a study’s methodology to enhance its implementation in a busy family medicine practice. The physician serving as committee chair is responsible for collecting committee member input and transmitting the final decision back to the AAFP NRN leadership.

Next, we assess the interest of the AAFP NRN membership. This process occurs before practice recruitment to validate that there is sufficient member interest in a particular study topic to meet recruitment goals. An AAFP NRN staff member communicates with network members who are appropriate for a given study, provides them with a 1-page overview of the study concept and a description of the responsibilities of the clinician or practice, and requests that they let us know if they are potentially interested in participating.

After a study idea progresses to this point in the process, the senior management begins developing a complete study protocol and pursuing funding options. Ultimately, it is the final decision of the AAFP NRN leadership and staff as to whether a project will proceed. However, because the success of network studies is contingent on participation of members, we strive to assure that they have ample input into the design of a research project early in the process.

The AAFP NRN developed its Principles for Industry-Funded Research (available at www.aafp.org/natnet) to guide negotiations with potential external commercial funders who invite the AAFP NRN to collaborate or take the lead on protocol development. This document outlines the principles by which the NRN collaborates with such industry sponsors, and clarifies the roles of both the AAFP NRN and the intended funders in the development, data collection, analysis, and dissemination of the research project.

Creating a Practical Budget

We begin to create a detailed budget once a preliminary protocol has been developed and potential funding source(s) have been identified. The Network Director and Research Administrator begin the budget process by reviewing the preliminary protocol and assigning costs by category, starting with personnel. Historically, we have tended to underestimate the amount of time required by staff to implement network studies. Therefore, we now pay particular attention to adequate salary coverage for the project manager and research assistants when possible. Once personnel costs have been determined, all other project-related costs are added into the budget. Our experience indicates that 2 other cost categories for PBRN studies require particular attention: study site training costs and payments to participating practices.

Depending on the funding source, a draft budget is either submitted for approval in the case of commercial and most private philanthropic funding or, in the case of government-sponsored funding, reviewed to determine whether the proposed budget fits into the funder’s allotment for the project. If the budget exceeds the available funds, the AAFP NRN director will make adjustments that ensure the project can be conducted without compromising the integrity of the study.

Composing the Study Team

The dynamics of the research team are essential to the success of any study. The composition of the team offers distinctive challenges to PBRN research. Members of the team generally include the principal investigator (PI), coinvestigators, project manager, consultants, statistician, research assistants, senior scientist(s), practicing physicians, and practice study coordinators. In addition, PBRNs carrying out prospective cohort studies, clinical trials, or practice change interventions must also consider sophisticated information management resources and formal linkages with the statistical and methodologic expertise existing in academic centers.⁷ An overall description of the study team’s roles appears in Table 1.

Practice-based research is distinctive because physicians and study coordinators from the selected practices are essential members of the study team. Their roles are critical in the planning stage of a research study. They provide invaluable feedback on the feasibility of implementing the protocol in practice. These practice representatives provide feedback on whether the proposed study protocol is feasible given their specific local requirements. Early engagement of these team members always improves implementation of the study protocol. The AAFP NRN first involves these practice leaders during the face-to-face study training session, as described in the following sections.

The AAFP NRN has developed policies and procedures to standardize network operations. These policies strengthen the understanding of the research team, expedite our work, and eliminate future misunderstandings. In addition, we have developed model agreements designed to ensure a common understanding between the AAFP NRN and outside research team members. All the policies and agreements listed below are available at www.aafp.org/natnet.

“Writing Process for Manuscripts” includes the roles, expectations, timelines, and procedures for writing papers related to studies conducted in the AAFP NRN. This writing process was developed to facilitate the timely dissemination of research findings to practices and to the academic press. This agreement states that the AAFP NRN expects to publish the results of every study conducted in our network.

The Publication Policy outlines how investigators working with the AAFP NRN are to prepare manuscripts for primary and secondary publication. It covers topics such as authorship, manuscript preparation and titles, conflict resolution, and acknowledging the AAFP NRN and study sites in published papers.

The External PI Agreement provides a detailed description of the relationship between the AAFP NRN and PIs who are not staff of the AAFP. It clarifies that any projects brought to the network by external PIs must be consistent with the AAFP NRN′s mission, objectives, and current research agenda. The document also outlines the development of project budgets, AAFP NRN policies for day-to-day project operations, financial responsibilities and authority of the AAFP NRN director, and expectations for the dissemination of findings.

The Data Sharing Agreement sets forth the terms, conditions, and obligations concerning data sharing between the AAFP NRN and any another network or person. It clarifies that if the AAFP NRN conducts a study resulting in a project dataset, then it owns such data.

The General Affiliation Agreement asserts that the AAFP NRN is interested in promoting collaborative arrangements with state and regional networks. It also delineates the conditions under which the AAFP NRN and regional networks can collaborate, including study selection, decision making, data collection, and IRB approval for collaborative studies.

Recruiting and Selecting Sites

Practice recruitment and selection is another challenge for PBRNs. The practice recruitment and selection strategy is developed by the central research team for each study based on the number of practices, physicians, staff, and/or patients needed, determined by the power analysis and augmented by expected attrition.^23–26 Once the target sample size has been set, AAFP NRN staff assess whether a sufficient number of practices can be recruited exclusively from the AAFP NRN membership based on the study’s requirements for particular patient and/or practice characteristics.

In situations where AAFP NRN members will not provide adequate numbers, a decision is made whether affiliated state or regional networks will be invited to participate. The AAFP NRN staff generally works with affiliate network directors or administrators to identify members of those networks who may be appropriate for the study.

Once the recruitment strategy is developed, the project staff initiates the recruitment process. The chronology of practice recruitment and selection from AAFP NRN practices is generally the following:

1. A broadcast recruitment E-mail is sent to all eligible AAFP NRN members announcing the study and briefly describing the eligibility criteria.

2. One week later, a personalized letter that includes responsibilities and timeline, a 1-page study overview, a Study Interest Form, and a postage-paid return envelope are sent to all eligible AAFP NRN members.

3. Two days later, another broadcast E-mail is sent to all eligible members announcing that recruitment letters and invitations for the study have been mailed.

4. Affiliated networks recruit their own practices to participate in a study.

One to 2 weeks later, if adequate numbers of AAFP NRN and affiliated network practices have not volunteered to participate in the study, we send individual E-mails and make personal telephone calls to selected AAFP NRN members inviting them to participate. In addition, we may contact staff of the AAFP regular electronic newsletter, AAFP News Now, about publishing a recruitment article in an upcoming edition. If necessary, we may contact additional affiliated networks about their participation in the study.

We select practices from the pool of eligible and interested practices that meet inclusion criteria and maximize diversity based on geographic location, practice type, size, and any other pertinent characteristics. We also consider other factors such as whether a practice is already participating in another study, and a practice and study coordinator’s previous experience in network studies (in particular, if we are selecting practices for a more complex study). We have created a database to assist us in practice selection in which we store these data on all AAFP NRN member practices, physicians, and study coordinators.

For each participating practice, the lead physician, study coordinator, Practice Signatory Official (as designated by the practice), and the study PI sign a Practice Agreement. This document states that the practice agrees to carry out all designated responsibilities of the study and the AAFP NRN agrees to provide materials, provide and pay for training, and provide the designated stipend. Institutional Review Board (IRB) issues are spelled out in the Unaffiliated Investigator Agreement.

At times during the recruitment phase we must make tough decisions about balancing scientific rigor with the realities of “real-world” medical practice. For example, occasionally we are forced to modify the initial sampling plan to achieve the required sample size. In a recent study, we initially planned to include few residency practices. However, an eligibility requirement that a practice either had to deliver at least 50 babies or perform 50 well-baby visits in the previous year forced us to change our original recruitment strategy and include residency practices to get sufficient numbers for our study. In addition, although in an ideal world we would have the luxury of randomly selecting practices to increase the representativeness of our sample, this is not always possible in PBRN studies. Sometimes we need to include all interested practices to reach sample size requirements. Finally, although a study design may require equal numbers of practices with or without a particular characteristic (eg, those with and without electronic medical records), we sometimes learn after a study begins that a practice, in fact, did not really have that characteristic. Thus, when the budget permits, we oversample practices just as we oversample patients. For instance, if a study is randomized at the practice level and the power analysis indicates that 10 practices per arm are sufficient, we plan for attrition by budgeting for 12 practices per arm.

Training Practice Staff and Physicians

The AAFP NRN has adopted a strategy of providing face-to-face training for practices selected to participate in a research study. When funding permits, we require each participating practice to send a team, usually comprising the site’s lead physician and study coordinator, for a weekend of training. Sessions generally run all day Saturday and, occasionally, through Sunday morning. In this way, staff and physicians are not out of the office during a busy work week. Bringing these people together ensures that the lead physicians and study coordinators understand the protocol and all participating practice staff receive the same instructions for implementing the study in the practice. This increases the likelihood that the study will be implemented with fidelity in all participating practices.

The entire research team, including all investigators, consultants, project manager, and research assistants, conducts the weekend sessions. The sessions have evolved into both a successful strategy for implementing studies in practices and a place where the practices make significant contributions that improve the design and feasibility of study protocols. Discussions encourage brainstorming on best strategies for implementing the protocol in a practice. As mentioned previously, practice leadership provides a new perspective, which frequently leads to improved protocol design and implementation, and ultimately to high quality study data.²⁷

Training sessions also provide the opportunity for site leaders and the research team to get to know one another. In contrast to many regional networks, our national staff has few opportunities to meet face-to-face with members. Training sessions may be the only time during the study period that the research team and practice staff meet. These personal connections between the practice members and the research team lead to better communication throughout the study period and eventually lead to a more successful research study.

When key personnel from study sites are unable to attend the face-to-face training, we provide telephone training or ask someone who did attend the training to orient those who did not. However, these techniques cannot replicate the experience of a face-to-face training weekend with all the practices.

In addition, we develop an in-depth training manual that includes copies of and instructions for completion of all forms; the AAFP IRB application; all consent forms; instructions for completion of human subjects training; contact information for the central research team, practice physicians, practice study coordinators, and other key people; the study protocol, as appropriate; suggestions for study implementation, including recruitment and consent process for subjects; and other pertinent information. The manual is the basis of the training curriculum and agenda. A template for the manual and schedule appears in Table 2.

The cost and logistic issues involved in planning training sessions has led to the development of some useful guidelines. In an effort to control expenses and simplify the process of making travel arrangements for the training sessions, we use a travel agency. All participants are asked to make their own travel arrangements through the agency within the provided parameters for arrival and departure times and maximum ticket price. In this way, we can also limit the number of participants who arrive late or leave early. In addition, all flight expenses are billed directly to the AAFP NRN, eliminating the inconvenience of reimbursing participants. This system has worked well for both participants and the research team.

After training, practice participants and the research staff are generally excited and motivated to implement the study. To preserve the momentum generated from the training, we try to minimize the time between training and implementation in practices. However, this is not always possible for reasons such as if numerous changes to the protocol are made as a result of feedback from training participants; if practices need to train their staff and physicians before study initiation; or if practice-specific implementation strategies need to be developed. In cases where there is an extended period of time between the training session and study implementation in the practice, we conduct one-on-one refresher telephone calls with the study coordinator or the lead physician before the start of data collection.

Conclusion

Although the nature of PBRNs presents numerous challenges to conducting quality studies, the AAFP NRN has identified a number of processes that facilitate the implementation of studies on a national level. These strategies include how to select fundable, feasible studies; compose the study team; recruit and select sites; and train practice staff and physicians. The planning procedures described in this paper can be modified to work for state or regional PBRNs, and can create a strong foundation on which to build rigorous PBRN studies. Proper planning for PBRN studies significantly improves the success of study implementation and the dissemination of scientifically sound results.