No data is data

Elżbieta M. Goździak reflects on silence, missing answers, and what counts as data.

“I'm not getting any data.”

I hear this kind of complaint often. I have heard versions of it from students and researchers for years. Interviewees provide short, monosylabic answers. Participants repeatedly say “I don't know.” People don't talk about something the researcher expected to be important.

My response has often been: No data is data! This of course does not mean that we can turn every silence into whatever interpretation suits us. It means that the absence of the expected material deserves methodological attention.

When Children Don't Talk

At the peak of the COVID pandemic, I chatted with a young researcher dismayed by the fact that children interviewed online frequently gave very short, sometimes monosyllabic answers. My interlocutor was quite frustrated by these outcomes: the interviews weren't generating sufficiently rich data.

However, let us examine the research encounter itself. The interviews were happening on a screen. In many cases, there was an interpreter mediating the conversation and and another adult might have also been present. The child was being questioned by an unfamiliar researcher under already strange pandemic circumstances.

I suggested that rather than asking: Why won't these children talk?, the interviewer should ask: What is it about this research encounter that isn't inviting them to talk?

The monosyllables don't tell us why automatically, but they tell us that there is something we need to understand. Instead of treating them simply as a failure to produce data, we might ask how the conditions of the interview contributed to producing precisely these kinds of answers.

Researchers don't simply collect data. Research encounters help produce the data we collect.

The Wrong Question

For years, I have been baffled by surveys that haven't been constructed on the basis of exploratory ethnography. Survey designers often devise categories that make perfect sense to them and then become puzzled when respondents repeatedly answer:

I don't know.

Neither.

Other.

Not applicable.

I often tell students to pay particular attention to those responses. If they occur repeatedly, perhaps respondents aren't failing to answer the survey; perhaps the survey is failing to describe their world. I also ask students whether they can see themselves in a survey instrument they are supposed to fill out. Is there actually an answer there that represents their experience?

There is another reason anthropologists should pay attention to the answers that don't fit neatly. Much social research is understandably interested in patterns: how many people give the same answer, how strongly they agree, whether a response is common enough to constitute a meaningful finding. Anthropologists are interested in patterns too, but we are also interested in range and variation. The person who says something nobody else says may not be statistical noise. The outlier may point us toward an experience, category, or way of understanding the world that our research design did not anticipate.

Consensus can tell us something. So can disagreement. So can the answer that doesn't fit any of the boxes we provided.

And agreement does not necessarily settle the matter. People may agree because they share an experience, but they may also share a vocabulary, an institutional explanation, or an understanding of what one is supposed to say.

What if the failure to answer our question is telling us something about our question?

Exploratory research helps us learn what categories people themselves use, what distinctions matter to them, and even what questions make sense to ask before we freeze our assumptions into a questionnaire.

Missing Expectations

Sometimes ethnographers themselves expect something to emerge and it simply doesn't.

We expect participants to talk about X and they talk about Y. Adults describe something as enormously important, children barely mention it, and teens just shrug their shoulders. A category central to the research literature turns out not to be a category interlocutors use themselves.

I have encountered this even in the language we use to describe research participants. Editors have sometimes preferred the term Latinx. Yet Latina mothers I interviewed did not use the term and, when asked about it, some did not know what it meant. They described themselves using other terms. This creates an interesting methodological dilemma. Do I use the category preferred in a particular academic or editorial context, or the language my interlocutors use to describe themselves?

For an anthropologist, that is not simply a question of terminology. The mismatch itself is data. It tells us something about the distance that can exist between academic categories and the social worlds those categories are intended to describe. Even a category adopted with the intention of being inclusive can become an imposed category if the people to whom we apply it neither use nor recognize it.

The same thing can happen with the questions we bring into the field. What seems important in the scholarly literature may barely register in people's accounts, while something we had treated as peripheral becomes central to their stories. That does not necessarily mean the literature is wrong or that our interlocutors have given us the final word. But the discrepancy between what we expected to hear and what people actually talk about is something to investigate rather than explain away.

An Empirical Vacuum

I learned another lesson about “no data” during my years of research on human trafficking of children and adolescents. There was certainly no shortage of writing about trafficking. Newspapers, reports, books, policy documents, and academic publications were filled with stories about trafficking, particularly trafficking for sexual exploitation, and trafficking of children. What was in surprisingly short supply was solid empirical research.

And yet the claims were often remarkably confident. Authors of newspaper articles, reports, and academic papers seemed to know how widespread trafficking was, who was being trafficked, how trafficking happened, and what should be done about it, even when the empirical basis for some of these assertions was very, very thin or non-existent.

Some studies relied heavily on small or highly selective samples. Others interviewed “key stakeholders.” There is nothing inherently wrong with interviewing stakeholders—I have done it myself—but we need to remember that stakeholders have a stake. They see the phenomenon from a particular advocacy and institutional position. Their knowledge is valuable, but it is not the same as an all-encompassing view of the phenomenon.

What struck me most was how easily an empirical vacuum could be filled with assumptions. Once repeated often enough, claims began to acquire the appearance of established facts.

This taught me an important qualification to my favorite methodological aphorism:

No data is data, but no data is not evidence.

Taking the absence of data seriously is not the same as filling that absence with speculation. Absence of evidence is not evidence of absence. But neither does it give us permission to make claims that the available evidence cannot support.

Sometimes the most accurate answer a researcher can give is simply:

We don't know.

Reading Absences

Silence, monosyllabic answers, missing categories, refusals, I don't know, and things people simply don't mention can all become objects of inquiry. But they require more, not less, methodological discipline.

We ask:

What conditions produced this response?

Did we ask a meaningful question?

Who else was present?

What relationship had we established?

What assumptions were built into our categories?

What were we expecting to hear, and why did we expect to hear it?

Would another method or setting produce something different?

Is this absence a finding, or does it tell us that we need to investigate further?

And, sometimes, are we simply asking people about something that matters much more to us than it does to them?

No Easy Answers

As I emphasized earlier, “No data is data” is not permission to interpret every silence. It is, however, a reminder not to discard what does not look like the data we hoped to collect. Sometimes an “I don't know” tells us something about the participant. Sometimes it tells us something about the question. And sometimes the absence of evidence tells us simply that we do not know.

And that last possibility may be the hardest one for researchers to accept.