Improving survey methodologies for cessation studies

Background

A primary outcome measure for our tobacco cessation evaluation clients is quitting success measured seven months after program registration. Program and funding decisions are informed by this important measure. In any study that uses self-reported survey results, the representativeness of the sample affects the accuracy and generalizability of the results. In studies that ask about the success of a program (such as quitting tobacco), those who do not respond to the survey are less likely to be successful.

Unfortunately, the past decade has seen response rates for most population surveys decreasing, and quit studies are no exception. PDA has been working for the past two decades to conduct studies with ample response rates, to address issues of representativeness, and to provide cost-effective survey solutions to our clients. 

Approach

We are always honing our craft. We attend conferences such as the American Association of Public Opinion Researchers and comb the literature for effective methods. Some of our key survey strategies include:

Experimenting with multi-mode survey designs. Ten years ago, our quit surveys were conducted exclusively by phone. We now reach out to participants via mail, email, and even text, depending on the population. We have conducted studies that show the web response option both reduces survey costs and reaches a population that might not answer the phone. We built our own web-based survey management tool that allows us to seamlessly conduct multi-mode surveys.

Testing different incentive strategies. We are well-versed in different incentive strategies, including both prepaid and promised incentives of varying amounts. We routinely conduct experiments on the effect of different incentive amounts on response rate and survey cost.

Modifying our survey communications. We painstakingly design pre-notification letters and email invitations in hopes of increasing their open rates by potential responders. We experiment with the timing and wording of our communications to determine which are most effective.

Tweaking our survey attempts. We have done studies to look at the impact of the number of attempts to reach a participant on representativeness. We also test out the best days and times to make survey contacts.

Exploring how to most accurately interpret survey data. The decline in both survey response and consent rates across the field has led us to conduct investigations into the impact of these declines on responder quit rates.  In 2015 we collaborated with clients in three different states to examine the impact of declining response rates by studying survey responders that were the most difficult to reach. We have also used Bayesian methods to conduct simulations of true quit rates after applying assumptions about those who did not respond. Finally, in 2018 we partnered with St. Olaf College math and statistics students to explore options for measuring error associated with quit rates.

Results

Our work makes our clients happy. Our tobacco cessation clients enjoy response rates that are consistently among the highest of their peers, and routinely above the target of 50% set for these studies. Because of our investigations and studies, we accurately estimate survey costs and consult our clients on which methods are cost saving and which increase representativeness.

Our work contributes to the field. Between 2001 and 2018, PDA employees have presented findings from our survey methodology studies at 18 conferences. Seven current PDA employees have presented at one or more of the following conferences:

•       National Conference on Tobacco or Health
•       American Association of Public Opinion Researchers
•       American Evaluation Association
•       North American Quitline Consortium
•       Minnesota Health Services Research Conference
•       Minnesota Evaluation Studies Institute

We have become known in the tobacco cessation field as experts in survey methodology and in 2009 and 2015 were asked by the North American Quitline Consortium to author an issue paper on best practices for calculating a quit rate. These NAQC papers provide recommendations for survey protocols, study eligibility, sample sizes, dealing with missing data, interview training, and data collection that are considered standards in the field. Our paper “The effect of survey nonresponse on quitline abstinence rates: Implications for practice” was published Nicotine and Tobacco Research in 2015