Survey Mode and Data Quality: A Look at Careless Responding Across Three Modes and Two Cultures

Much psychological research depends on participants’ diligence in filling out materials such as tests or surveys. However, not all participants are motivated to respond attentively, which leads to unintended issues with the quality of the data. Our question is: how do different modes of data collection - paper/pencil, web/computer-based, and smart-phone - effect participants’ diligence vs. “careless responding” tendencies and, thus, the data quality? Results from prior studies suggest that different modes of data collection produce a comparable prevalence of careless responding tendencies. However, as technology develops and data are collected with increasingly diverse populations, this question needs to be readdressed, also by looking at cultural differences. The present research examined the effect of survey mode on careless responding across three waves in a repeated-measures design. Following recommendations in the literature, we computed a careless responding index as a composite of six indicators that capture aspects of a participant’s inattentiveness. In a sample of working adults from China (N=78), we found that participants were significantly more careless when completing the survey materials in web/computer-based than in paper/pencil mode. In a sample of German students (N=61), participants were significantly more careless when completing the paper/pencil mode compared to the smart-phone or web/computer mode. This talk will discuss why these results deviate from past findings that investigated study mode and hypothesize about potential cross-cultural differences.