Understanding Society Scientific Conference 2019 - , University of Essex

Do we know what to do with 'Don't know'?

My Programme   Add 
Day:Thu 4 Jul
  • Luke Taylor, Kantar Public

An experiment was conducted on the UK’s Understanding Society Innovation Panel (IP11) comparing three treatments of "Don’t know" (DK) response codes within a self-completion (CAWI and CASI) questionnaire.

The three treatments were:

1. The approach currently used on Understanding Society: ‘hiding’ DK codes and only making them available where respondents try to move on without selecting an answer

2. As above but with a specific prompt at each question on how to view additional option

3. Including DK codes as part of the main response lists (so they are always visible).

Our analysis compares the three treatments focusing on overall response distributions, question completion times and differences by mode. Open data were analysed to add further context.

Analysis was conducted across 26 questions. Treatment 3 consistently elicited a higher proportion of DK responses than treatment 1. The differences at self-assessed health measures tended to be small (typically in the order of one or two percentage points). The differences were particularly pronounced for attitudinal questions on low-salience issues.

When asked about the benefits and risks of nuclear energy, only 9% of those exposed to treatment 1 answered DK, compared with 18% for treatment 2 and 31% for treatment 3. Our analysis suggests that treatment 3 effectively discourages the reporting of ‘non-attitudes’. Under treatment 1, 64% of those that reported knowing ‘nothing at all’ about nuclear energy provided a valid (non-DK) answer at the benefits and risks question; this fell to 46% for treatment 2, and 26% for treatment 3.