Effects of data collection technique on the quality of data: A MTMM study of CATI, CAPI and Online interviews

Annette Scherpenzeel (CentERdata, Tilburg University)

In this study, a split-ballot mtmm design is used to compare the quality of the data collected by web interviews with the quality of the data collected by traditional data collection methods.
A probability sample of the Dutch population was contacted and interviewed by one of two traditional computer assisted data collection modes: telephone interviewing or face-to-face interviewing.

At the end of these interviews, the respondents were asked to become a member of an online household panel. Respondents who were willing to participate but had no internet access were equipped with a user-friendly computer with internet access. All interviewed respondents who joined the panel were asked to fill in the same interview in a web interview format, a few months after the first telephone of face-to-face measurement.

The mtmm analysis gives estimates of both reliability as the complement of random error variance and validity as the complement of systematic method variance for every measure in the study. These estimates show that panel data collected using a self-administered web interview are at least as valid and reliable as data collected using more traditional modes of interviewing, when the sample is kept constant.