Using representativeness indicators to evaluate the impacts of non-response on Understanding Society survey dataset quality
|Day:||Thu 4 Jul|
We assess the impacts of non-response on Understanding Society survey dataset quality. Non-response is problematic in surveys because non-respondent – respondent differences can cause estimate non-response biases compared to fully observed values. Understanding Society is a major UK longitudinal survey on social, economic and health topics. In longitudinal surveys, the impacts of non-response on datasets across survey waves (i.e. sample attrition) are of interest, as are impacts during within wave data collection: some subjects are only interviewed after multiple attempts, so cost conscious designers must decide how many attempts to make to obtain acceptable dataset quality. To measure dataset quality, we use representativeness indicators (Coefficients of Variation of response propensities: CVs), which quantify subset-sample similarity in terms of variation in inclusion propensities estimated given an auxiliary covariate set. Low levels of variation imply low risks of biases in subset estimates. We assess how the Understanding Society dataset changes across waves compared to the design and non-response weighted first wave dataset (other information on first wave non-respondents is not available), including computing partial CV variants to quantify auxiliary covariate associated impacts on datasets that could be targeted by method modifications to improve dataset quality. We also analogously consider (with wave) data collection, with a focus on whether datasets stabilise after fewer than the current number of attempts to interview subjects. Then, using our findings, we advise on future Understanding Society data collection.