Cooking Data with Care: The Role of Contextual Inquiry in Large-Scale Quantitative Research
Date:
Talk at eScience Institute, University of Washington, Seattle, Washington
Abstract: How do we bring human-centered perspectives and cultural contexts to data-intensive, highly-automated algorithmic ways of knowing? Qualitative and quantitative methods are often imagined as orthogonal ways of answering questions in fundamentally different, incommensurable ways. In this talk, I argue that there is often substantial qualitative contextual inquiry and expertise deployed in quantitative methods. Such insights are crucial to “cooking data with care,” as Geoff Bowker advocated. These situated knowledges are typically implicitly leveraged as analysts make key decisions about what data to use, when data needs to be calibrated, how to transform and merge data for a given purpose, how to reduce dimensionality to work with complex datasets, which variables to use as proxies, and how to interpret the validity of results. To illustrate, I share experiences with a recent publication in which we directly integrated ethnographic and qualitative methods to reproduce, extend, and ultimately contest the interpretations of a previous large-scale computational study that claimed to discover substantial levels of conflict between automated bots in Wikipedia. Our mixed-methods approach is an integrative, iterative synthesis, not a coordinated pluralism: there is a column in a dataframe that would not exist without ethnography, as well as thick descriptions of cases that were found by sampling from the extreme ends of a statistical distribution. Through my experiences in this project, I discuss several lessons learned and future directions, which touch on topics including open science, reproducibility, and data ethics.