Tar Wars and Smoking Prevention: Premature Claim of Effectiveness

To the Editor: Tar Wars, as evaluated by Cain et al,¹ is part of a large movement of knowledge-based, early-education programs designed to inform elementary school students about the dangers of tobacco use. The authors have attempted to lend support to the claim that “school-based tobacco education is one part of a comprehensive approach to youth tobacco prevention activities that has been shown to decrease youth tobacco initiation.” They cite several articles that describe existing programs. Their intervention seems to be quite similar to programs shown to be effective, which raises questions about the authors’ claim that Tar Wars contributes anything uniquely effective. We respectfully question the methods presented by the authors and would like to make several suggestions in the hope that future assessments of the Tar Wars program might yield substantive evidence of effectiveness.

One of our concerns is that students in the Cain evaluation study were pre- and posttested after only 1 presentation; moreover, the full Tar Wars program with its various components was not included for testing. Considering that authors of previous studies of tobacco education programs waited 1 to 2 years to test whether information persisted to determine effectiveness, the significance of this short-tem test is questionable. The test tapped into whether information would be remembered by students after only 9 to 11 days, and students were aware of the information on the pretest when they heard the presentation. This does not seem to represent an enduring test of knowledge or attitude change. We would suggest that the Tar Wars posttest be conducted after 1 year.

A second concern is that the quantitative measure used in the evaluation included only 14 items, and among them were 3 potentially confusing or irrelevant items about tobacco advertisements in the mass media. If we eliminated these 3 as too ambiguous, the entire program effectiveness would be based on 11 questions, most of the answers to which were already known by the students. Of 14 questions, students on average answered 8 or 9 correctly on the pretest. Therefore, of the 11 unambiguous questions, there are only 2 previously unknown questions on which to gauge the effectiveness of the entire program. The authors should consider reformatting the questionnaire by adding questions and ensuring that the questions are clear and relevant.

The authors conclude that the students participating in the program indeed learned new information. Results showed an increase in correct responses for all questions. However, because of the methodology, the increase may potentially be explained by a practice effect. If children saw the questionnaire before the presentation and recognized the questions a few days later after the presentation, we fear that the increase in correct answers doesn't say much about the actual effectiveness of the intervention or about attitude change, only that they remembered test items they might have missed. Perhaps this could be remedied if the measures occurred further apart in time.

The authors acknowledge the lack of a control group in this effectiveness study. Students’ knowledge after participating in the program was not compared with other students who did not receive the program intervention. However, the authors claim that other agents of change to explain the children's acquisition of knowledge are “unlikely” and insist that “major changes [to the program] are not needed.” It would seem that the authors have no research foundation on which to make this claim. The lack of a control group with which to confirm the gain of knowledge combined with the questionable methods used substantially weaken the authors’ claim that no improvements are necessary to the program.

Effectiveness cannot be attributed to Tar Wars if much of the program is repetition of previously known information. Although Tar Wars may eventually prove to be a valuable addition to school curricula, at this time it is unlikely that it has added anything unique or novel beyond similar programs.