Talking about scientific results without overstating the findings

Sometimes the most productive conversations at scientific conferences happen outside the formal sessions. That was the case for Psychology Professor Maureen Callanan, who attended a meeting a few years ago during which she participated on a panel about research on diversity in developmental psychology.

Afterwards, she and her colleague Susan Gelman, a professor of psychology and linguistics at the University of Michigan, discussed researchers' tendency to take a simple finding and state it as a universal truth, generalizing beyond the scope of an individual study.

"It's the difference between saying, 'What the 5-year-olds in this study did…' and 'What 5-year-olds do,'" explained Callanan, a developmental psychologist. "To imply that all children behave a certain way ignores that most research in developmental psychology is based on studies of white, middle-class children."

This use of "generic language" gives the impression there's no nuance, and it can be misleading, Callanan and Gelman agreed. And they started to wonder how prevalent it is in the literature.

Fast forward to August 26, when Callanan and Gelman published an article they coauthored with Jasmine DeJesus and Graciela Solis in the Proceedings of the National Academy of Sciences, "Generic Language in Scientific Communication." Their analysis of nearly 1,150 psychology journal articles published in 2015-16 revealed that generic language was used in 89 percent of the research summaries. The use of such "universal" language is particularly troublesome given that 73 percent of those articles made no mention of the race or ethnicity of their participants, one of several demographic factors that could limit the relevance of findings about one group to another; similarly, 79% of articles did not mention the income level of the sample, and 74% didn't note language background.

"Using generic language is especially problematic if researchers are overgeneralizing from selective or limited samples," said Callanan. The authors noted that the field of psychology even has an acronym that acknowledges the homogeneous nature of typical samples: WEIRD, which stands for participants from Western, educated, industrialized, rich, and democratic societies.

The authors identify several forces contributing to the use of generalizations, including a desire to make the work accessible to broad audiences—and appealing to funding agencies. However, glossing over findings and failing to describe study samples could lead to exaggerated conclusions, they caution.

"Because scientists are encouraged to reach broad conclusions, their writing may sacrifice precision in favor of bolder claims," said DeJesus, assistant professor at the University of North Carolina at Greensboro and the article's first author.

A follow-up study indicated research summaries that used generic language pack more punch than more nuanced summaries, at least among the undergraduates and others survey participants, who consistently rated them "more important."

"The challenge is figuring out how to report findings in a way that's honest and also gets the work noticed," said Callanan, who speculated that this phenomenon may also occur in fields other than psychology. "I'd like to see researchers do a better job of being accessible and concise, while also fully acknowledging variability and the limits of our evidence."

At a minimum, Callanan urged researchers to refrain from making sweeping generalizations and to consider referring to their participants' behavior in the past tense to emphasize that the findings are limited to the study and not necessarily applicable to other groups or a predictor of future behavior.

"A single study really isn't enough basis for a broad generalization," she said. "That type of language might be more appropriate when reviewing a whole literature and summarizing across multiple studies."

Even with a representative sample, it may not be appropriate to generalize, noted Callanan, who has wrestled with these issues in her own work. "I'm less focused on findings that can be generalized, because there is always nuance," she said. "There are always differences, even among white, middle-class kids."

Callanan also sees broader negative consequences of using generic language to talk about scientific findings that are based on samples from very specific groups—often middle-class, white families. "It makes people think that's the norm, and if findings among another group are different, it is considered a deficit," she said. "That's the danger."

Rather, it's important to study individual communities on their own terms, not as outliers or exceptions to the norm, she said. Avoiding generic language is part of a broader effort to be inclusive, an effort that has shaped Callanan's work over three decades. Her own research focuses on how children learn about science and the world around them, and she appreciates the tremendous range and variety she has observed.

"People have different ways of learning about the natural world," she said. "I'm not looking anymore for the one way children learn. I think that puts us on a false path. We should be looking not just for generalizable findings but also for variation, because there is going to be both in whatever topic we're studying."