The importance of using qualitative techniques to explore mental models in survey design

Introduction 

Mental models are important concepts behind how respondents interact with surveys. They are based on an individual’s knowledge and experiences. They affect how an individual conceptualises topics and respond to survey questions. By designing to match a respondent's mental model, we can reduce friction and improve both experience and data quality. 

Goals and Methods 

This project was undertaken by the Office for National Statistics (ONS) to transform questions on the Labour Force Survey, on accidents and illness in the workplace. 

The insights required to understand mental models were gathered using an innovative qualitative research method; Cogability testing. These sessions combine traditional cognitive interviewing with emerging usability practices to explore both comprehension and practical response. Respondents were observed completing the question section. Researchers concurrently and retrospectively probed on aspects of comprehension and usability. 

Results 

An open coding method was used to examine the transcripts in depth. Annotations and concepts (codes) were applied to short sentences to identify ideas, actions, perceptions and relevance (in vivo codes). Codes were then grouped into themes such as; ease of the survey, understanding of the question, response options perceptions, and usability. 

Results demonstrated that the current questionnaire flow did not match the respondents’ mental models. For example, respondents did not delineate between their accidents and illnesses, resulting in them confusing concepts and answering incorrectly. This resulted in reduced data quality and the potential for mixed measurement. 

Conclusions 

The research enabled a redesign of the questionnaire using evidence-based insights. The Cogability method was paramount in gaining these, and in ensuring the final design was effective. Subsequently the design aligned with respondent mental models, resulting in an optimal respondent experience and improved quality throughout the data lifecycle.