Web survey paradata on response time outliers

A systematic literature review

Abstract

In the last two decades, survey researchers have intensively used computerised methods for the collection of different types of paradata, such as keystrokes, mouse clicks and response times, to evaluate and improve survey instruments as well as to understand the survey response process. With the growing popularity of web surveys, the importance of paradata has further increased. Within this context, response time measurement is the prevailing paradata approach. Papers typically analyse the time (measured in milliseconds or seconds) a respondent needs to answer a certain item, question, page or questionnaire. One of the key challenges when analysing the response time is to identify and separate units that are answering too quickly or too slowly. These units can have a poor response quality and are typically labelled as response time outliers. This paper focuses on approaches for identifying and processing response time outliers. It presents a systematic overview of scientific papers on response time outliers in web surveys. The key observed characteristics of the papers are the approaches used, the level of time measurement, the processing of response time outliers and the relationship between response time and response quality. The results show that knowledge on response time outliers is scattered, inconsistent and lacking systematic comparisons of approaches. Consequently, there is a need to improve and upgrade the knowledge on this issue and to develop new approaches that will overcome existing deficiencies and inconsistencies in identifying and dealing with response time outliers.