Patent citations could signal which biomedical papers lead to a real-world impact

Of the hundreds of thousands of scientific papers published every year, it’s not always clear which ones will make a real-world impact, whether in the realm of improving treatments or supporting new research tools, like high-powered microscopes.

Now, scientists at Stanford Medicine have developed a system to help measure the tangible impact of biomedical papers.

The research team created a metric that relies on patent citations. Scientists who file a patent, which marks the transition of an idea or finding to an actual product or process, must list scientific papers from which their idea drew. The references to papers are known as patent citations.

The researchers opted to track patent citations as a way to measure the impact of a scientific manuscript, given patents often serve as a foundation for new companies or products. They reasoned that papers cited in a patent are the ones most likely to inform real-world change and innovation.

“We all know that many breakthroughs in the clinic and in technology come from basic findings or innovations in the lab, but the process by which one actually translates that science to the real world is opaque,” said Ishan Kumar, a graduate student at Stanford. The team hopes its new metric will not only help connect the dots but will provide a new lens through which translational productivity at universities and institutions can be characterized.

Zou and his colleagues also found that racially and ethnically diverse research teams were significantly more likely to publish papers that were cited in patents. They based their diversity assessments on name analysis. Additionally, the data showed that papers with a female lead or senior author were less likely to be listed in a patent citation.

A paper describing the study was published today in Nature Biotechnology. Kumar and James Zou, PhD, professor of biomedical data science, are co-senior authors. Stanford medical student Anoop Manjunath is the lead author.

The proof is in the patent

It’s been difficult to decipher which scientific discoveries will move from the lab to the clinic or into commercial use, as well as the demographic characteristics of those whose work is likely to make the transition, Zou said. Typically, the impact of scientific papers is measured through something called a research-citation factor, which measures the number of times other scientists refer to a certain paper in their own work. There are inherent limitations with that approach: A researcher may be more inclined to cite papers from colleagues or friends, for instance, and citations do not capture the full extent of their paper’s influence — but it still provides some indication of a paper’s impact.

Zou and his team didn’t set out to replace this barometer, but rather to supplement it with something traditional citation factors do not measure.

“It’s incredibly difficult — but important — to measure translation and innovation,” Zou said. “Once we can do that, it opens doors to measure other things, too, like how federal funding affects scientific discovery and innovation at universities.”

It’s incredibly difficult — but important — to measure translation and innovation.

In devising their new method of evaluation, the team aggregated 2.4 million scientific publications from PubMed and 125 million patents filed in the Google Patent Public Datasets from 1970 to 2015. Tey then created an algorithm to link the research with the patents. In the study, papers cited in a patent were seen as more directly connected to the translation of ideas into practice.

After parsing the data, the team saw that papers from biomedical research powerhouses such as the University of California system, Johns Hopkins University and Stanford University ranked high in patent citations. But they also saw that some smaller institutions that produce fewer papers overall, such as the Broad Institute, also ranked in the top tier, suggesting that smaller, specialized organizations can have just as much, if not more, of a translational impact per publication when it comes to biomedical innovations.

Diversity and gender effects

While analyzing the data for additional insights, two trends stood out to Zou and the team: Papers with a female first or senior author were less likely to be cited in a patent. What was particularly surprising, said Manjunath, was that the gender gap persisted during the 45-year period they examined. “We know that in many other academic fields, there certainly has been a demographic change, especially over the last few years, but that hasn’t really been reflected in the pattern of citations,” he said. That may be, in part, due to the longer lag time in patent citations, Manjunath said, adding that perhaps the gender gap in patent citations will close in the coming years.

The second trend that caught the team’s attention was that publications with more racially and ethnically diverse authors were significantly more likely to be cited in a patent when compared with papers with a less diverse group of authors. This trend persists after accounting for the scientific field.

“We all have lots of experiences working with other scientists from different countries who have different backgrounds and ideas. And I think we’re intuitively aware that having more diverse teams improves the quality of research and the innovativeness of teams overall,” Zou said. “But what’s really nice about the analysis Anoop and Ishan did was they actually measured innovation outcomes as it correlates with diversity. That’s interesting, even if it’s just correlative at this point.”

In other words, the analysis can’t prove that the diversity of the teams is what caused the higher number of patent citations, but the findings suggest a link between diversity and innovation, Zou said.

Manjunath and Kumar are seeking out some of the reasons the gender gap in patent citations appears to be so persistent. The team is conducting interviews with scientists — from masters of patent filing to those without a patent to their name — to better understand the process and why some papers might be overlooked.

“My hope is that the feedback we get will help inform how to make changes, perhaps at a policy level, so that there are equitable opportunities to be rewarded for one’s work,” Kumar said.

Research associate Arya Gowda is a Stanford co-author on the paper.

Researchers from Carnegie Mellon University, University of Illinois, Columbia University and Chan-Zuckerberg Biohub contributed to the study.

The study was funded by the National Science Foundation, the Silicon Valley Foundation and the Chan Zuckerberg Initiative.