Measurement Practices in UX Research: A Systematic Quantitative Literature Review

Abstract

User experience research relies heavily on survey scales as an essential method for measuring users' subjective experiences with technology. However, repeatedly raised concerns regarding the improper use of survey scales in UX research and adjacent fields call for a systematic review of current measurement practice. Until now, no such systematic investigation on survey scale use in UX research exists. We, therefore, conducted a systematic literature review, screening 707 papers from CHI 2019 to 2021, of which 207 were eligible empirical studies using survey scales. Results show that papers frequently lacked rationales for scale selection (70.05%) and rarely provided all scale items used (21.74%). Nearly half of all scales were adapted (44.69%), while only one-fifth of papers reported any sort of scale quality investigation (19.32%). Furthermore, we identified 224 different scales and 287 distinct constructs measured. Most scales were used once (76.79%), and most constructs were measured once (81.88%). Results highlight questionable measurement practices in UX research and suggest opportunities to improve scale use for UX-related constructs. We provide recommendations to promote improved rigor in following best practices for scale-based UX research. This research was funded internally. All materials are available on OSF: https://osf.io/dgq6b/

Publication
PsyArXiv

Related