Abstract
Background: Existing health literacy assessments developed for research purposes have constraints that limit their utility for clinical practice, including time requirements and administration protocols. The Brief Health Literacy Screen (BHLS) consists of 3 self-administered Single-Item Literacy Screener (SILS) questions and obviates these clinical barriers. We assessed whether the addition of SILS items or the BHLS to patient demographics readily available in ambulatory clinical settings reaching underserved patients improves the ability to identify limited health literacy.
Methods: We analyzed data from 2 cross-sectional convenience samples of patients from an urban academic emergency department (n = 425) and a primary care clinic (n = 486) in St. Louis, Missouri. Across samples, health literacy was assessed using the Rapid Estimate of Adult Literacy in Medicine-Revised (REALM-R), Newest Vital Sign (NVS), and the BHLS. Our analytic sample consisted of 911 adult patients, who were primarily female (62%), black (66%), and had at least a high school education (82%); 456 were randomly assigned to the estimation sample and 455 to the validation sample.
Results: The analysis showed that the best REALM-R estimation model contained age, sex, education, race, and 1 SILS item (difficulty understanding written information). In validation analysis this model had a sensitivity of 62%, specificity of 81%, a positive likelihood ratio (LR+) of 3.26, and a negative likelihood ratio (LR−) of 0.47; there was a 28% misclassification rate. The best NVS estimation model contained the BHLS, age, sex, education and race; this model had a sensitivity of 77%, specificity of 72%, LR+ of 2.75, LR− of 0.32, and a misclassification rate of 25%.
Conclusions: Findings suggest that the BHLS and SILS items improve the ability to identify patients with limited health literacy compared with demographic predictors alone. However, despite being easier to administer in clinical settings, subjective estimates of health literacy have misclassification rates >20% and do not replace objective measures; universal precautions should be used with all patients.
Health literacy, often defined as the degree to which individuals can obtain, process, and understand basic health information and services needed to make appropriate health decisions,1 is a critical predictor of health knowledge, health outcomes, and health care utilization.1,2 Limited health literacy has been associated with a higher rate of hospitalization,3⇓⇓–6 lower use of preventive services,5 and less effective management of chronic conditions.7 The translation of health literacy measurement beyond the research environment to clinical settings in order to help target potential interventions has been hampered by tools that require administration by staff and face other barriers to completion.8⇓–10 For example, the S-TOFHLA is timed and can take up to 7 minutes to complete, increasing the potential for interruptions that could affect performance.11
When considering implementation of health literacy assessments in overcrowded and understaffed medical settings, researchers must consider the trade-offs between instrument complexity, patient acceptability, and diagnostic accuracy.12,13 If found to be brief, accurate, and reliable, health literacy screening instruments could be converted to iPad/kiosk applications that patients could complete while awaiting care, as has been done for dementia,14 vision,15 and substance abuse.16 The Brief Health Literacy Screen (BHLS) contains 3 Single Item Literacy Screener (SILS) items, self-administered, brief, subjective questions through which patients report their perceived health literacy skills, avoiding some of the barriers presented by objective screening tools. The diagnostic accuracy and validity of the SILS relative to the Rapid Estimate of Adult Literacy in Medicine (REALM) and Newest Vital Sign (NVS) have been previously reported.11,17⇓–19
In prior research, the BHLS has been validated to detect limited health literacy using the S-TOFHLA as the criterion standard in a study of 332 white veterans (area under the receiver operating characteristics curve [AUROC], 0.76–0.87).18 The BHLS was subsequently validated in a large Veterans Administration patient population (n = 1796) of mostly older white men with at least a high school education. The “confident with forms” item performed the best, and the ability to identify patients with limited health literacy varied based on the reference standard (AUROC, 0.74 for S-TOFHLA, 0.84 for REALM). In a subsequent study, Wallace et al20 evaluated the 3 SILS items using the REALM as the criterion standard in a population (n = 305) consisting of predominantly white women with a mean age of 49.5 years. “Confident with forms” was superior to the other questions and demographic information (sex, age, race, educational attainment, health insurance). The ability to identify limited health literacy (AUROC, 0.82) on REALM was similar to that determined by Chew et al.17
In several clinical studies, associations have found between SILS and various health outcomes.2,21⇓⇓–24 Limited health literacy measured using SILS has been shown to be associated with discontinuation of antidepression medication among patients with type 2 diabetes,25 perception of low coordination of care and low satisfaction among women with breast cancer,26 health care discrimination among diabetics,27 increased risk of hospital admissions,5 decreased knowledge of chronic disease among hypertensive and diabetic patients,3 poorer physical and mental health among older adults,28 and poorer outcomes among diabetic patients.21 In addition, the BHLS has been validated for use in clinical settings when administered by nurses during patient intake.24
However, age, race, and education, which can be readily collected in clinical settings, were found to be significant predictors of health literacy in a systematic review of 85 studies.29 Therefore, there is a need to examine the ability of SILS items and the BHLS, in addition to demographic factors, to identify patients with limited health literacy.20 We quantitatively assessed whether the addition of each SILS item or the BHLS improves the ability to identify patients with limited health literacy compared with patient demographic information. We hypothesized that a combination of the SILS and demographic characteristics improves the ability to identify patients with limited health literacy, compared with standard sociodemographic variables, in clinical settings where administration of objective health literacy assessments is not feasible.
Methods
Settings and Participants
We analyzed data from 2 cross-sectional convenience samples of patients from an emergency department (ED) (n = 425) and primary care clinic (n = 486) affiliated with an urban academic medical center in St. Louis, Missouri. Using SAS statistical software (SAS Inc., Cary, NC), half of the participants from each sample were randomly assigned to the estimation data set, and the remaining observations were combined to form the validation data set.
Emergency Department
Trained research assistants recruited patients between March 1, 2011, and February 29, 2012, from an urban academic ED. Patients aged ≥18 years were identified for enrollment by review of the electronic medical record dashboard. Exclusion criteria included undue patient distress as judged by the attending physician, altered mental status, aphasia, mental handicap, previously diagnosed dementia or insurmountable communication barrier as judged by family or the screener, non-English-speaking, sexual assault victims, acute psychiatric illness, or corrected visual acuity worse than 20/100 using both eyes. This study was approved by the hospital institutional review board. Research assistants administered health literacy assessments to all eligible and consenting patients and recorded their responses. Demographic data were collected during the interview and from the electronic medical record. De-identified age, race, and sex data were recorded for patients declining to participate. A total of 588 patients were approached; 139 (24%) refused, 9 were excluded, and 446 (76%) were enrolled. Enrolled patients' age, sex, and race did not significantly differ from patients who refused to participate or from the ED patient population.11,30,31
Primary Care Clinic
Participants were recruited between July 2013 and April 2014 from the Primary care clinic (PCC) of the same large, urban academic medical center. Patients in the waiting rooms of the PCC were approached by trained data collectors and asked to complete a survey in English. Inclusion criteria were that participants be at least 18 years old, a patient at the PCC, and speak English. Participants were asked to complete a self-administered written questionnaire and a verbally administered survey component. The latter component assessed health literacy with the REALM, Revised (REALM-R) and NVS and was administered by a trained data collector, who recorded responses. All participants completed a verbal consent process and signed a written consent form before completing the survey. This study was approved by the Human Research Protection Office at Washington University School of Medicine.
Approximately 26% (n = 1111) of those approached were ineligible to participate in the study because they were not patients, did not speak English, or had previously taken the survey. Among eligible participants, 44% (n = 1380) agreed to participate in the study and gave consent to trained data collectors. Of the 1380 patients who gave consent, 975 (71%) completed the written survey. Among those with complete written surveys, 602 (60%) completed the verbally administered component. Survey respondents were generally similar to the underlying primary care clinic patient population with respect to sex, age, and race.
For inclusion in this analysis, participants must have completed all 3 health literacy assessments (ie, the REALM-R, NVS, and BHLS) and have demographic data (age, sex, race, education). Because of the small number of patients in the “other race” category for both the ED (n = 11) and PCC (n = 27) samples, we limited analysis to patients whose self-reported race was white or black and who met all inclusion criteria (n = 425 for the ED and n = 486 for the PCC).
Health Literacy Assessments
REALM, Revised
The REALM-R is a health literacy assessment (word recognition test) in which participants are asked to pronounce 11 common medical terms: fat, flu, pill, allergic, jaundice, anemia, fatigue, directed, colitis, constipation, and osteoporosis. The first 3 words are included to reduce test anxiety and are therefore not scored as part of the REALM-R. A trained REALM-R administrator scores the pronunciation (correct/incorrect) of each of the remaining 8 words, resulting in 8 possible points.8 Using standard scoring, we dichotomized the REALM-R score into limited health literacy (scores 0 to 6) and adequate health literacy (scores >6).32
Newest Vital Sign
The NVS is a verbally administered, 6-item measure that asks about information contained in a standard food nutrition label, which requires reading comprehension and numeracy skills.33 Participants received an NVS score ranging from 0 to 6 based on the number of correct answers. Scores from 0 to 1 reflect a high likelihood of limited health literacy; 2 to 3, a possibility of limited health literacy; and 4 to 6, adequate health literacy.33 For analysis, NVS was dichotomized as limited health literacy (scores 0–3) and adequate health literacy (scores 4–6).
Brief Health Literacy Screen
Participants were administered 3 written SILS items, which were measured on 5-point Likert scales that assess self-reported health literacy skills: “How often do you have problems learning about your medical condition because of difficulty understanding written information?” (1 = always, 2 = often, 3 = sometimes, 4 = rarely, 5 = never); “How confident are you filling out medical forms by yourself?” (1 = not at all, 2 = a little bit, 3 = somewhat, 4 = quite a bit, 5 = extremely confident); and “How often do you have someone help you read hospital materials?” (1 = always, 2 = often, 3 = sometimes, 4 = rarely, 5 = never). In the estimation models, these questions were dichotomized into limited health literacy (responses <4) or adequate health literacy (responses ≥4) as individual predictors and continuously as a BHLS sum score, based on prior studies.17,18,34
Statistical Analysis
Sample characteristics for the overall combined samples (N = 911) and the estimation (n = 456) and validation (n = 455) samples are examined to ensure no demographic differences between samples. Five estimation models for 2 validated objective health literacy measures (REALM-R, NVS) are compared. We started with a base multivariable logistic regression model consisting of patient demographic information; age (continuous); sex (female, male); race (white, black); and education (less than high school, high school diploma or equivalent degree, more than high school). Categorical variables were modeled using indicators, with male as the reference for sex, white as the reference for race, and high school (middle category) as the reference level of education. Each SILS item is examined individually by adding them one at a time to the base model; these models are compared with a model that includes the full BHLS sum score. To select a final estimation model we used 3 goodness-of-fit criteria: rescaled R2, Akaike information criterion (AIC), and AUROC. R2 and AUROC values closer to 1 and smaller AIC values are obtained from models with better fit. Data were analyzed using SAS software version 9.4 (SAS, Inc.); statistical significance was assessed at P < .05.
Based on the best estimation model, we estimated the probability of limited health literacy for each participant in the validation sample. The limited health literacy cutoff was determined by the lowest misclassification rate to establish an ideal trade-off between sensitivity and specificity. We examined the discrimination (ability to distinguish patients with limited health literacy from those with adequate health literacy) of the final estimation model and the cutoff selected by examining concordance (sensitivity, specificity) using a 2 × 2 table, kappa statistic (and 95% confidence interval [CI]), and misclassification rate. The kappa statistic measures interrater agreement; we examined the agreement between the estimation models and validated objective health literacy assessments (REALM-R, NVS) for determining patients with limited health literacy.35,36 We assessed this model as a diagnostic test for limited health literacy by calculating positive and negative likelihood ratios (LR+ and LR−, respectively).
Results
The analytic sample consisted of 911 patients; the majority were women (62%), black (66%), and had at least a high school education (83%). Patient age ranged from 18 to 94 years, with an average age of 49 years (standard deviation, 14 years). The majority of patients were assessed as having adequate health literacy based on the REALM-R (54%) but limited health literacy according to the NVS (63%). The majority (72%) reported “rarely” or “never” having difficulty understanding written information. More than half of the patients reported being “extremely” or “quite a bit” confident (62%) when filling out medical forms. A majority (74%) stated that they “rarely” or “never” have someone help them read hospital materials. Half of this sample was randomly selected to the estimation sample (n = 456) and the other half to the validation sample (n = 455); there were no significant differences in sex, education, age, race and health literacy as assessed by the REALM-R, NVS, BHLS, or SILS between the estimation and validation samples based on the 2-sample test for proportions (sex, education, race, REALM-R, NVS, SILS) and 2-sample t test (BHLS, age) (see Table 1).
REALM-R Estimation
Table 2 presents the model results and goodness-of-fit statistics for 5 REALM-R estimation models. All demographic predictors, with the exception of age, were statistically significant in the base model that contained demographic predictors only (age, education, sex, and race); the goodness-of-fit statistics suggested a model with fair estimation ability (R2 = 0.34; AIC = 505; AUROC = 0.79). Addition of the “difficulty with written information” SILS created a model that identified limited health literacy (R2 = 0.38; AIC = 491; AUROC = 0.81) better than the base model. Models containing the 2 other SILS items, or the BHLS, did not identify patients with limited health literacy as well.
NVS Estimation
All demographic predictors, except sex, were statistically significant in the base model that contained demographic predictors only; the goodness-of-fit statistics suggested a model with fair estimation ability (R2 = 0.20; AIC = 535; AUROC = 0.73). Addition of the “difficulty with written information” SILS with demographics identified patients with limited health literacy (R2 = 0.23; AIC = 525; AUROC = 0.75) better than the demographics-only model. Models containing the 2 other SILS items did not identify patients with limited health literacy as well. The full BHLS model had slightly better estimation (R2 = 0.24; AIC = 524; AUROC = 0.75) than the 1 SILS item model (Table 3).
Validation
Using model coefficients and lowest misclassification cutoffs, the validation sample was used to compare the estimation of limited health literacy by the models with the “difficulty with written information” SILS and the BHLS with both objective health literacy assessments (REALM-R, NVS).
Difficulty Understanding Written Information (SILS)
The addition of the “difficulty with written information” SILS item to demographic information (age, sex, race, education) has the ability to identify limited health literacy on the REALM-R, with a sensitivity of 62%, specificity of 81%, a 28% misclassification rate, and a moderate kappa statistic of 0.43 (95% CI, 0.35–0.51).37 The likelihood ratio of a positive test result (LR+) is 3.26, and the likelihood ratio of a negative test (LR−) is 0.47; this model slightly underestimates (39%) limited health literacy in the sample (Table 4). This model showed greater sensitivity (82%) and lower specificity (68%) in estimating the NVS, attenuating the LR+ (2.56) and improving the LR− (0.26). The NVS estimation model also had a lower misclassification rate (24%) and a slight increase in kappa statistic to 0.49 (95% CI, 0.41–0.57); this model estimates limited health literacy among 63% of the sample.
Brief Health Literacy Screen
The addition of the BHLS to demographic information has the ability to estimate limited health literacy on the REALM-R, with a sensitivity of 80%, specificity of 62%, an LR+ of 2.11, an LR− of 0.32, a 30% misclassification rate, and a moderate kappa statistic of 0.42 (95% CI, 0.34–0.50).37 This model estimates 58% limited health literacy in the sample, overestimating limited health literacy (Table 4). The BHLS estimation model had slightly lower sensitivity (77%) and higher specificity (72%) for estimating the NVS; improving the LR+ (2.75) and preserving LR− (0.32). The NVS estimation model also had a lower misclassification rate (25%) and a slight increase in kappa statistic to 0.48 (95% CI, 0.40–0.56). This model estimates limited health literacy among 59% of the sample, underestimating limited health literacy (Table 4).
Discussion
The utility of SILS items and the BHLS in clinical practice have been demonstrated;18,38,39 we extend this work to examine predictive ability compared with and combined with demographic characteristics that can be easily collected in clinical settings. Age, sex, race, education, and 1 SILS item (difficulty understanding written information) were found to be predictors of limited health literacy; combined they yielded the best estimation model for limited health literacy measured by the REALM-R and NVS. This model identified patients with limited health literacy better than demographic factors alone. We posit that differences between the results of our analyses and previous studies could be attributed to sample demographics and analysis techniques. Our sample included only English speakers and was predominately nonwhite (69% black). We used regression analytic approaches and assessed 2 objective measures of health literacy (REALM-R and NVS), as well as multiple predictors of limited health literacy, in both ED and primary care settings. Most previous studies have examined only 1 objective measure of health literacy among patients in only 1 clinical setting (primary care), and do not report likelihood ratios to facilitate the clinical interpretation of these health literacy screening test results.19,38,40 The extension of this work to the ED has important implications because the majority of rural EDs are staffed by family medicine physicians.41⇓–43
BHLS estimation models have slightly higher misclassification rates than the “difficulty understanding written information” SILS estimation models for both REALM-R and NVS, suggesting that the use of 1 SILS item in addition to demographic information can improve the ability to identify limited health literacy in fast-paced clinical settings serving medically underserved populations. Despite being easier to administer in clinical settings, however, SILS subjective measures of health literacy have misclassification rates of >20% when used in addition to known demographic predictors and do not replace objective measures.
Our study has several limitations that should be considered when interpreting the findings. This is a convenience sample of English-speaking ED and primary care patients at a single urban academic medical center, and analysis was limited to black and white respondents because of the small number of patients from other racial/ethnic groups, limiting the generalizability of findings to other populations. As with most health literacy measures, SILS items do not assess oral communication, listening, writing,44 or visual literacy,45 and do not consider age, sex, language, culture, education, health condition, and health care settings.46 While we did see variability in health literacy, most of the sample had at least a high school education, and we excluded those with visual impairments from our study because the health literacy measures are not validated for this population.
While the NVS can be performed in <3 minutes11,40 this still requires staff to administer the test, and so it is not feasible in many clinical settings. There has been some work to examine the feasibility of a self-administered NVS, but the instrument has yet to be validated.47 In this study we validated our limited health literacy estimation model against 2 validated objective health literacy measures (REALM-R and NVS).
Conclusions
Our findings endorse the utility of 1 SILS question combined with demographics to identify patients with limited health literacy in fast-paced clinical settings, rather than objective assessments that may not be feasible. Future research is needed to refine these models and predictors that decrease misclassification rates and to examine the validity of this approach in other populations. It is important to note that, given the high misclassification rates, universal precautions should be considered for use in all patients.48,49
Acknowledgments
The authors acknowledge the assistance of our research and screening staff: Lucy D'Agostino McGowan, William D. MacMillan, Renee Gennarelli, Meng-Ru Cheng, Sarah Lyons, Nhi Nguyen, Ralph O'Neil, Emma Dwyer, Ian Ferguson, Mallory Jorif, Matthew Kemperwas, Jasmine Lewis, Darain Mitchell, Margaret Lin, Andrew Melson, and John Schneider.
Notes
This article was externally peer reviewed.
Funding: MSG is supported by the Barnes-Jewish Hospital Foundation; Siteman Cancer Center; National Institutes of Health (NIH); National Cancer Institute (grant nos. P30 CA91842, U54CA153460, R01 CA168608, 3U54CA153460-03S1, U54 CA155496); the Patent-Centered Outcomes Research Institute (grant ID 4586); Department of Defense (grant no. W81XWH-14-1-0503); and the Washington University School of Medicine (WUSM) and WUSM Faculty Diversity Scholars Program. RTG is supported by an institutional KM1 Comparative Effectiveness Award (no. KM1CA156708) through the National Cancer Institute (NCI) at the NIH; and the Clinical and Translational Science Award (CTSA) program of the National Center for Research Resources and the National Center for Advancing Translational Sciences at the NIH (grant nos. UL1 RR024992, KL2 RR024994, TL1 RR024995). RTG is also supported through the Emergency Medicine Foundation/Emergency Medicine Patient Safety Foundation Patient Safety Fellowship. CRC is supported by the National Institute of Aging (grant no. 1U13AG048721-01) and by the Washington University Emergency Care Research Core, which receives funding from the Barnes-Jewish Hospital Foundation. KAK is supported by the Washington University School of Medicine; Barnes-Jewish Hospital Foundation; Siteman Cancer Center; the National Cancer Institute (grant nos., R01 CA168608, 3U54CA153460-03S1, and P50CA95815); the National Institute of Diabetes and Digestive and Kidney Diseases (grant P30 DK092950); the Agency for Health care Research and Quality (grant no. R21 HS020309); and the Centers for Disease Control and Prevention (grant no. U58 DP0003435).
Conflict of interest: none declared.
Disclaimer: The content of this article is solely the responsibility of the authors and does not necessarily represent the official views of the supporting societies and foundations or the funding agencies.
- Received for publication January 26, 2015.
- Revision received April 16, 2015.
- Accepted for publication April 29, 2015.
References
In this issue
Jump to section
Related Articles
Cited By...
- Health literacy and cumulative social disadvantage are associated with survival and transplant in patients with hepatocellular carcinoma: a prospective study
- Questionnaire validation practice within a theoretical framework: a systematic descriptive literature review of health literacy assessments
- Practice-based Research Networks (PBRNs) in the United States: Growing and Still Going After All These Years