Abstract
Purpose
Interest in collecting patient-reported outcomes (PROs), such as health-related quality of life (HRQOL), health status reports, and patient satisfaction is on the rise and practical aspects of collecting PROs in clinical practice are becoming more important. The purpose of this paper is to draw the attention to a number of issues relevant for a successful integration of PRO measures into the daily work flow of busy clinical settings.
Methods
The paper summarizes the results from a breakout session held at an ISOQOL special topic conference for PRO measures in clinical practice in 2007.
Results
Different methodologies of collecting PROs are discussed, and the support needed for each methodology is highlighted. The discussion is illustrated by practical real-life examples from early adaptors who administered paper–pencil, or electronic PRO assessments (ePRO) for more than a decade. The paper also reports about new experiences with more recent technological developments, such as SmartPens and Computer Adaptive Tests (CATs) in daily practice.
Conclusions
Methodological and logistical issues determine the resources needed for a successful integration of PRO measures into daily work flow procedures and influence significantly the usefulness of PRO data for clinical practice.
Similar content being viewed by others
References
Greenhalgh, J., Long, A. F., & Flynn, R. (2005). The use of patient reported outcome measures in routine clinical practice: Lack of impact or lack of theory? Social Science and Medicine, 60, 833–843. doi:10.1016/j.socscimed.2004.06.022.
Li, C., Friedman, B., Conwell, Y., & Fiscella, K. (2007). Validity of the patient health questionnaire 2 (PHQ-2) in identifying major depression in older people. Journal of the American Geriatrics Society, 55, 596–602. doi:10.1111/j.1532-5415.2007.01103.x.
Ouimette, P., Wade, M., Prins, A., & Schohn, M. (2008). Identifying PTSD in primary care: Comparison of the primary care-PTSD screen (PC-PTSD) and the general health questionnaire-12 (GHQ). Journal of Anxiety Disorders, 22, 337–343.
Williams, J. W., Jr, Noel, P. H., Cordes, J. A., Ramirez, G., & Pignone, M. (2002). Is this patient clinically depressed? Journal of the American Medical Association, 287, 1160–1170. doi:10.1001/jama.287.9.1160.
Juniper, E. F., Bousquet, J., Abetz, L., & Bateman, E. D. (2006). Identifying ‘well-controlled’ and ‘not well-controlled’ asthma using the asthma control questionnaire. Respiratory Medicine, 100, 616–621. doi:10.1016/j.rmed.2005.08.012.
Atherton, P. J., & Sloan, J. A. (2006). Rising importance of patient-reported outcomes. The Lancet Oncology, 7, 883–884. doi:10.1016/S1470-2045(06)70914-7.
Bren, L. (2006). The importance of patient-reported outcomes…it’s all about the patients. FDA Consumer, 40, 26–32.
van Andel, G., Bottomley, A., Fossa, S. D., Efficace, F., Coens, C., Guerif, S., et al. (2008). An international field study of the EORTC QLQ-PR25: A questionnaire for assessing the health-related quality of life of patients with prostate cancer. European Journal of Cancer, 44, 2418–2424. doi:10.1016/j.ejca.2008.07.030.
Chisholm, M. A., Williamson, G. M., Lance, C. E., & Mulloy, L. L. (2007). Predicting adherence to immunosuppressant therapy: A prospective analysis of the theory of planned behaviour. Nephrology, Dialysis, Transplantation, 22, 2339–2348. doi:10.1093/ndt/gfm149.
Luszczynska, A., Sarkar, Y., & Knoll, N. (2007). Received social support, self-efficacy, and finding benefits in disease as predictors of physical functioning and adherence to antiretroviral therapy. Patient Education and Counseling, 66, 37–42. doi:10.1016/j.pec.2006.10.002.
Efficace, F., Innominato, P. F., Bjarnason, G., Coens, C., Humblet, Y., Tumolo, S., et al. (2008). Validation of patient’s self-reported social functioning as an independent prognostic factor for survival in metastatic colorectal cancer patients: Results of an international study by the Chronotherapy Group of the European Organisation for Research and Treatment of Cancer. Journal of Clinical Oncology, 26, 2020–2026. doi:10.1200/JCO.2007.12.3117.
Gotay, C. C., Kawamoto, C. T., Bottomley, A., & Efficace, F. (2008). The prognostic significance of patient-reported outcomes in cancer clinical trials. Journal of Clinical Oncology, 26, 1355–1363. doi:10.1200/JCO.2007.13.3439.
Rumsfeld, J. S., MaWhinney, S., McCarthy, M., Jr., Shroyer, A. L., VillaNueva, C. B., O’Brien, M., et al. (1999). Health-related quality of life as a predictor of mortality following coronary artery bypass graft surgery. Participants of the Department of veterans affairs cooperative study group on processes, structures, and outcomes of care in cardiac surgery. Journal of the American Medical Association, 281, 1298–1303. doi:10.1001/jama.281.14.1298.
Gwaltney, C. J., Shields, A. L., & Shiffman, S. (2008). Equivalence of electronic and paper-and-pencil administration of patient-reported outcome measures: A meta-analytic review. Value in Health, 11, 322–333. doi:10.1111/j.1524-4733.2007.00231.x.
Bettinville, A., Rabenberg, H., & Hansgen, K. D. (2005). An enquiry into the WILDE-Intelligence-Test (WIT): Comparability of application of the paper-pencil-version vs. the computer based application—an analysis based on data of the Leipzig Vocational Retraining Center. Die Rehabilitation, 44, 237–243. doi:10.1055/s-2005-866926.
Folk, L. C., March, J. Z., & Hurst, R. D. (2006). A comparison of linear, fixed-form computer-based testing versus traditional paper-and-pencil-format testing in veterinary medical education. Journal of Veterinary Medical Education, 33, 455–464. doi:10.3138/jvme.33.3.455.
Norman, G. J., Sallis, J. F., & Gaskins, R. (2005). Comparability and reliability of paper- and computer-based measures of psychosocial constructs for adolescent physical activity and sedentary behaviors. Research Quarterly for Exercise and Sport, 76, 315–323.
Webb, P. M., Zimet, G. D., Fortenberry, J. D., & Blythe, M. J. (1999). Comparability of a computer-assisted versus written method for collecting health behavior information from adolescent patients. The Journal of Adolescent Health, 24, 383–388. doi:10.1016/S1054-139X(99)00005-1.
Heuser, J., & Geissner, E. (1998). Computerized version of the pain experience scale: A study of equivalence. Schmerz (Berlin, Germany), 12, 205–208. doi:10.1007/s004820050143.
Velikova, G., Wright, E. P., Smith, A. B., Cull, A., Gould, A., Forman, D., et al. (1999). Automated collection of quality-of-life data: A comparison of paper and computer touch-screen questionnaires. Journal of Clinical Oncology, 17, 998–1007.
Cook, A. J., Roberts, D. A., Henderson, M. D., Van Winkle, L. C., Chastain, D. C., & Hamill-Ruth, R. J. (2004). Electronic pain questionnaires: A randomized, crossover comparison with paper questionnaires for chronic pain assessment. Pain, 110, 310–317. doi:10.1016/j.pain.2004.04.012.
Schaeren, S., Bischoff-Ferrari, H. A., Knupp, M., Dick, W., Huber, J. F., & Theiler, R. (2005). A computer touch-screen version of the North American Spine Society outcome assessment instrument for the lumbar spine. The Journal of Bone and Joint Surgery. British Volume, 87, 201–204. doi:10.1302/0301-620X.87B2.15548.
Bliven, B. D., Kaufman, S. E., & Spertus, J. A. (2001). Electronic collection of health-related quality of life data: Validity, time benefits, and patient preference. Quality of Life Research, 10, 15–22. doi:10.1023/A:1016740312904.
Burke, J. D., Burke, K. C., Baker, J. H., & Hillis, A. (1995). Test-retest reliability in psychiatric patients of the SF-36 Health Survey. International Journal of Methods in Psychiatric Research, 5, 189–194.
Kleinman, L., Leidy, N. K., Crawley, J., Bonomi, A., & Schoenfeld, P. (2001). A comparative trial of paper-and-pencil versus computer administration of the Quality of Life in Reflux and Dyspepsia (QOLRAD) Questionnaire. Medical Care, 39, 181–189. doi:10.1097/00005650-200102000-00008.
Ryan, J. M., Corry, J. R., Attewell, R., & Smithson, M. J. (2002). A comparison of an electronic version of the SF-36 General Health Questionnaire to the standard paper version. Quality of Life Research, 11, 19–26. doi:10.1023/A:1014415709997.
Saleh, K. J., Radosevich, D. M., Kassim, R. A., Moussa, M., Dykes, D., Bottolfson, H., et al. (2002). Comparison of commonly used orthopaedic outcome measures using palm-top computers and paper surveys. Journal of Orthopaedic Research, 20, 1146–1151. doi:10.1016/S0736-0266(02)00059-1.
Wilson, A. S., Kitas, G. D., Carruthers, D. M., Reay, C., Skan, J., Harris, S., et al. (2002). Computerized information-gathering in specialist rheumatology clinics: An initial evaluation of an electronic version of the Short Form 36. Rheumatology (Oxford England), 41, 268–273. doi:10.1093/rheumatology/41.3.268.
Beebe, T. J., Harrison, P. A., McRae, J. A., & Evans, J. A. (2006). The effects of data collection mode and disclosure on adolescent reporting of health behavior. Social Science Computer Review, 24, 476–488. doi:10.1177/0894439306288690.
DeAngelis, S. (2000). Equivalency of computer-based and paper-and-pencil testing. Journal of Allied Health, 29, 161–164.
Duncan, P., Reker, D., Kwon, S., Lai, S. M., Studenski, S., Perera, S., et al. (2005). Measuring stroke impact with the Stroke Impact Scale: Telephone versus mail administration in veterans with stroke. Medical Care, 43, 507–515. doi:10.1097/01.mlr.0000160421.42858.de.
Hepner, K. A., Brown, J. A., & Hays, R. D. (2005). Comparison of mail and telephone in assessing patient experiences in receiving care from medical group practices. Evaluation & the Health Professions, 28, 377–389. doi:10.1177/0163278705281074.
de Vries, H., Elliott, M. N., Hepner, K. A., Keller, S. D., & Hays, R. D. (2005). Equivalence of mail and telephone responses to the CAHPS Hospital Survey. Health Services Research, 40, 2120–2139. doi:10.1111/j.1475-6773.2005.00479.x.
Powers, J. R., Mishra, G., & Young, A. F. (2005). Differences in mail and telephone responses to self-rated health: Use of multiple imputation in correcting for response bias. Australian and New Zealand Journal of Public Health, 29, 149–154. doi:10.1111/j.1467-842X.2005.tb00065.x.
Beebe, T. J., Mcrae, J. A., Harrison, P. A., Davern, M. E., & Quinlan, K. B. (2005). Mail surveys resulted in more reports of substance use than telephone surveys. Journal of Clinical Epidemiology, 58, 421–424. doi:10.1016/j.jclinepi.2004.10.007.
Kraus, L., & Augustin, R. (2001). Measuring alcohol consumption and alcohol-related problems: Comparison of responses from self-administered questionnaires and telephone interviews. Addiction (Abingdon, England), 96, 459–471. doi:10.1046/j.1360-0443.2001.9634599.x.
Ravens-Sieberer, U., Erhart, M., Wetzel, R., Krugel, A., & Brambosch, A. (2008). Phone respondents reported less mental health problems whereas mail interviewee gave higher physical health ratings. Journal of Clinical Epidemiology, 61, 1056–1060. doi:10.1016/j.jclinepi.2007.12.003.
Rodriguez, H. P., von Glahn, T., Rogers, W. H., Chang, H., Fanjiang, G., & Safran, D. G. (2006). Evaluating patients’ experiences with individual physicians: A randomized trial of mail, Internet, and interactive voice response telephone administration of surveys. Medical Care, 44, 167–174. doi:10.1097/01.mlr.0000196961.00933.8e.
Krueger, A. B., & Stone, A. A. (2008). Assessment of pain: A community-based diary survey in the USA. Lancet, 371, 1519–1525. doi:10.1016/S0140-6736(08)60656-X.
Meara, A. (2008). Director of medical management at the CMO of the Montefiori Medical Center. Ref Type: Personal Communication.
Bischoff-Ferrari, H. A., Vondechend, M., Bellamy, N., & Theiler, R. (2005). Validation and patient acceptance of a computer touch screen version of the WOMAC 3.1 osteoarthritis index. Annals of the Rheumatic Diseases, 64, 80–84. doi:10.1136/ard.2003.019307.
Cook, I. A., Balasubramani, G. K., Eng, H., Friedman, E., Young, E. A., Martin, J., et al. (2007). Electronic source materials in clinical research: Acceptability and validity of symptom self-rating in major depressive disorder. Journal of Psychiatric Research, 41, 737–743. doi:10.1016/j.jpsychires.2006.07.015.
Kable, S., Henry, R., Sanson-Fisher, R., Ireland, M., & Cockburn, J. (2006). Is a computer questionnaire of childhood asthma acceptable in general practice? Family Practice, 23, 88–90. doi:10.1093/fampra/cmi079.
Thumboo, J., Wee, H. L., Cheung, Y. B., Machin, D., Luo, N., & Fong, K. Y. (2006). Development of a smiling touchscreen multimedia program for HRQoL assessment in subjects with varying levels of literacy. Value in Health, 9, 312–319. doi:10.1111/j.1524-4733.2006.00120.x.
Williams, C. A., Templin, T., & Mosley-Williams, A. D. (2004). Usability of a computer-assisted interview system for the unaided self-entry of patient data in an urban rheumatology clinic. Journal of the American Medical Informatics Association, 11, 249–259. doi:10.1197/jamia.M1527.
Carr, A. J., Thompson, P. W., & Kirwan, J. R. (1996). Quality of life measures. British Journal of Rheumatology, 35, 275–281. doi:10.1093/rheumatology/35.3.275.
Bezjak, A., Ng, P., Skeel, R., DePetrillo, A. D., Comis, R., & Taylor, K. M. (2001). Oncologists’ use of quality of life information: Results of a survey of eastern cooperative oncology group physicians. Quality of Life Research, 10, 1–13. doi:10.1023/A:1016692804023.
Till, J. E. (1994). Measuring quality of life: Apparent benefits, potential concerns. The Canadian Journal of Oncology, 4, 243–248.
Taylor, K. M., Macdonald, K. G., Bezjak, A., Ng, P., & DePetrillo, A. D. (1996). Physicians’ perspective on quality of life: An exploratory study of oncologists. Quality of Life Research, 5, 5–14. doi:10.1007/BF00435963.
Katon, W., Von Korff, M., Lin, E., Walker, E., Simon, G. E., Bush, T., et al. (1995). Collaborative management to achieve treatment guidelines. Impact on depression in primary care. Journal of the American Medical Association, 273, 1026–1031. doi:10.1001/jama.273.13.1026.
Katz, S. J., Kessler, R. C., Lin, E., & Wells, K. B. (1998). Medication management of depression in the United States and Ontario. Journal of General Internal Medicine, 13, 77–86. doi:10.1046/j.1525-1497.1998.00022.x.
Regier, D. A., Narrow, W. E., Rae, D. S., Manderscheid, R. W., Locke, B. Z., & Goodwin, F. K. (1993). The de facto US mental and addictive disorders service system. Epidemiologic catchment area prospective 1-year prevalence rates of disorders and services. Archives of General Psychiatry, 50, 85–94.
Rose, M., Hess, V., Horhold, M., Brahler, E., & Klapp, B. F. (1999). Mobile computer-assisted psychometric diagnosis. Economic advantages and results on test stability. Psychotherapie, Psychosomatik, Medizinische Psychologie, 49, 202–207.
Greenhalgh, J., & Meadows, K. (1999). The effectiveness of the use of patient-based measures of health in routine practice in improving the process and outcomes of patient care: A literature review. Journal of Evaluation in Clinical Practice, 5, 401–416. doi:10.1046/j.1365-2753.1999.00209.x.
Espallargues, M., Valderas, J. M., & Alonso, J. (2000). Provision of feedback on perceived health status to health care professionals: A systematic review of its impact. Medical Care, 38, 175–186. doi:10.1097/00005650-200002000-00007.
Donaldson, M. S. (2004). Taking stock of health-related quality-of-life measurement in oncology practice in the United States. Journal of the National Cancer Institute. Monographs, 33, 155–167. doi:10.1093/jncimonographs/lgh017.
Detmar, S. B., Muller, M. J., Schornagel, J. H., Wever, L. D., & Aaronson, N. K. (2002). Health-related quality-of-life assessments and patient-physician communication: A randomized controlled trial. JAMA: The Journal of the American Medical Association, 288(23), 3027–3034. doi:10.1001/jama.288.23.3027.
Rubenstein, L. V., McCoy, J. M., Cope, D. W., Barrett, P. A., Hirsch, S. H., Messer, K. S., et al. (1995). Improving patient quality of life with feedback to physicians about functional status. Journal of General Internal Medicine, 10, 607–614. doi:10.1007/BF02602744.
Velikova, G., Booth, L., Smith, A. B., Brown, P. M., Lynch, P., Brown, J. M., et al. (2004). Measuring quality of life in routine oncology practice improves communication and patient well-being: A randomized controlled trial. Journal of Clinical Oncology, 22, 714–724. doi:10.1200/JCO.2004.06.078.
Ware, J. E., Snow, K. K., Kosinski, M., & Gandek, B. (1993). SF-36® Health Survey Manual and Interpretation Guide. Boston, MA: New England Medical Center, The Health Institute.
Kroenke, K., Spitzer, R. L., & Williams, J. B. (2002). The PHQ-15: Validity of a new measure for evaluating the severity of somatic symptoms. Psychosomatic Medicine, 64, 258–266.
Kroenke, K., Strine, T. W., Spitzer, R. L., Williams, J. B., Berry, J. T., & Mokdad, A. H. (2008). The PHQ-8 as a measure of current depression in the general population. Journal of Affective Disorders (in press).
Kroenke, K., Spitzer, R. L., & Williams, J. B. (2003). The patient health questionnaire-2: Validity of a two-item depression screener. Medical Care, 41(11), 1284–1292. doi:10.1097/01.MLR.0000093487.78664.3C.
McHorney, C. A., & Tarlov, A. R. (1995). Individual-patient monitoring in clinical practice: Are available health status surveys adequate? Quality of Life Research, 4, 293–307. doi:10.1007/BF01593882.
Embretson, S., & Reise, S. P. (2000). Item response theory for psychologists. Mahwah, NJ: Laurence Erlbaum Associates Inc.
Hambleton, R. K., Swaminathan, H., & Rogers, H. J. (1991). Fundamentals of item response theory. London: Sage Publications.
Embretson, S. E. (1996). The new rules of measurement. Psychological Assessment, 8, 341–349. doi:10.1037/1040-3590.8.4.341.
Wainer, H., Dorans, N., & Flaugher, R. (2000). Computerized adaptive testing: A primer. Hillsdale, NJ: Lawrence Erlbaum Associates.
Wainer, H., & Mislevy, R. J. (2000). Item response theory, item calibration, and proficiency estimation. In H. Wainer, N. J. Dorans, R. Flaugher, B. F. Green, R. J. Mislevy, L. Steinberg, & D. Thissen (Eds.), Computerized adaptive testing: A primer (2nd ed., pp. 61–101). Hillsdale, NJ: Lawrence Erlbaum Associates.
Rose, M., Bjorner, J., Becker, J., Fries, J. F., & Ware, J. (2008). Development of a preliminary physical function item bank supported the expected advantages of the Patient-Reported Outcome Measurement System (PROMIS). Journal of Clinical Epidemiology, 61, 17–33. doi:10.1016/j.jclinepi.2006.06.025.
Bjorner, J. B., Chang, C. H., Thissen, D., & Reeve, B. B. (2007). Developing tailored instruments: Item banking and computerized adaptive assessment. Quality of Life Research, 16(Suppl 1), 95–108. doi:10.1007/s11136-007-9168-6.
Cella, D., Yount, S., Rothrock, N., Gershon, R., Cook, K., Reeve, B., et al. (2007). The Patient-Reported Outcomes Measurement Information System (PROMIS): Progress of an NIH Roadmap cooperative group during its first two years. Medical Care, 45, S3–S11. doi:10.1097/01.mlr.0000258615.42478.55.
Reeve, B. B., Hays, R. D., Bjorner, J. B., Cook, K. F., Crane, P. K., Teresi, J. A., et al. (2007). Psychometric evaluation and calibration of health-related quality of life item banks: Plans for the Patient-Reported Outcomes Measurement Information System (PROMIS). Medical Care, 45, S22–S31. doi:10.1097/01.mlr.0000250483.85507.04.
Author information
Authors and Affiliations
Corresponding author
Appendix
Appendix
There are an increasing number of vendors that provide ePRO assessments on PDAs, PC or via IVR for clinical trails. The authors of this paper have personal experience with the systems briefly described below. We do not have any financial interest in any of the companies listed.
The following two start-up companies established themselves solely specialized in the electronic assessment of PRO within clinical practice (and outside clinical trails). First is Cibait Inc. (www.cibait.de), located in Germany serving approximately 40 clinics within Europe since 2002. The Cibait software supports Internet assessment, as well as assessments using PDAs, or tablet-PCs as stand alone devices. They also provide CATs for routine use. Their devices work as stand-alone versions as well as connected to a central server.
The second is Dynamic Clinical System (DCS, www.dynamicclinical.com), which is a spinoff company from the development efforts described in this paper at the Dartmouth-Hitchcock Medical Center. Today, DCS provides its service mainly to orthopedic clinics, serving almost 40 differing departments in around 20 institutions the U.S. They currently offer their main mode of administration via the Internet, with patient’s access from home or table computer in the clinic, which needs to be connected to the Internet via WLAN.
CQ-Office is a standalone product developed at the Charité in Berlin. It provides a highly customizable solution for standard questionnaires administered on a PDA platform (Linux or Windows Mobile operation system). The software is shareware, freely available but must be maintained by the user (for more information e-mail psychosomatik@uke.de).
One of the earliest ePRO assessment tools to our knowledge is the Quality of Life Recorder (www.ql-recoder.com) developed in Munich, Germany over 20 years ago. There have been many modifications. The software is free for students, but the systems can also be obtained as a commercial solution with the software preinstalled on a special hardware similar to a small tablet-PC.
We also like to mention one software solution called Chronorecord (www.chronorecord.org) as an example for a specific use of PRO assessments for a particular disorder. The software is developed to monitor patients with bipolar disorder.
Rights and permissions
About this article
Cite this article
Rose, M., Bezjak, A. Logistics of collecting patient-reported outcomes (PROs) in clinical practice: an overview and practical examples. Qual Life Res 18, 125–136 (2009). https://doi.org/10.1007/s11136-008-9436-0
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11136-008-9436-0