Quantifying the effect of mode on your analysis
|Day:||Tue 2 Jul|
Understanding Society now gives participants the option of completing the survey questionnaire online via their web browsers. This potentially impacts on the results of any data analysis because the distribution of the survey variables can vary depending on the mode used to administer a survey.
We begin by discussing more formally what is meant by a mode effect, why there are many possible mode effects, and why experiments are needed to estimate such effects. We then review and critique the available methods for adjusting for mode effects with a focus on those based on ‘instrumental variable’ techniques, before going on present the general approach we adopt here.
To illustrate what we did, we present analyses of data from the mixed-modes experiment carried out for Wave 8 of Understanding Society. We describe two examples in which we estimate the mode effects, interpret what these effects mean for data quality, and show the difference that using web instead of face-to-face made to these analyses. We finish by describing a simple way – using a random subsample of participants who were interviewed face-to-face - for users to check whether they need to worry about the effect of web mode, and what to do if one is found.