Bibliometrics

Bibliometrics is a quantitative field with connections to sociology of science, science policy, and research program evaluation. As data, bibliometrics typically measure research outputs, especially articles in peer-reviewed journals, often in terms of publication counts, citation counts, and measurements derived from these (such as publications per year or journal impact factors, which are roughly citations per article over a certain period of time). Other kinds of research outputs, such as patents and technology transfer, as well as various “alternative metrics” (or “altmetrics” like social media mentions or Wikipedia citations), are sometimes included as bibliometrics data. Bibliometric methods include various descriptive analyses of these data, as well as analytical methods such as statistical regressions, social network analysis (based on relationships such as coauthorships or shared citation patterns), text mining, and agent-based modeling.

Some researchers and analysts prefer the broader term “scientometrics,” reserving the term bibliometrics for simpler counts of publications and citations. Like other forms of relatively novel, large-scale social data, bibliometrics and scientometrics can be extremely powerful, but must be gathered and used thoughtfully.

This Toolkit includes important background reading to know before you start a bibliometrics project, a description of the most common sources for gathering bibliometrics data, some local resources, and references for further reading.

Background and caution

Hicks and Melkers (2012) discuss the general use of bibliometrics in program evaluation. They pay special attention to the interpretation of citation counts, emphasizing that “most [bibliometricians] agree that citation counts do not signify scientific quality in any simple way” (page 7). Mingers and Leydesdorff (2015) provide a substantial review of the scholarly bibliometrics literature, including discussions of various data sources, purported “laws” of bibliometrics, and formal definitions of various derived metrics such as the h-index (which attempts to measure both the productivity and citation impact of a scholar by measuring their most cited publications).

The impact factor and h-index are especially notorious as poorly-understood but widely-used metrics. The PLoS Medicine Editors (2006) provide a brief and critical explanation of the impact factor. Both impact factor and h-index are based on citation counts. It is widely recognized that citation practices differ dramatically across fields of research: biologists, mathematicians, and sociologists do not cite in the same ways. Consequently, it is also widely recognized that citation counts and derived statistics must be normalized for cross-field comparisons to be valid. However, Lee (2018) argues that a given article or body of research does not necessarily belong to a single field; this implies that the reference class used for normalizing citation counts is not well-defined. Specifically, a normalization calculation that is appropriate for one analysis might not be appropriate for another analysis.

Simple interpretations of bibliometrics might also be confounded by social hierarchies in science. A large-scale cross-disciplinary, interdisciplinary analysis by Elsevier (2017) found that women tend to have lower publication counts than men, but have similar normalized citation rates. However, some studies of specific fields have found gender disparities in citation rates (Maliniak et al. 2013; Dion et al. 2018; Fox and Paine 2019). Other potential confounders might include race and ethnicity, country, career status and position type, and institution type (e.g., research university vs. teaching-focused liberal arts college or community college).

In light of these kinds of challenges with appropriately using and interpreting bibliometrics, especially the impact factor, leaders of the bibliometrics community published two documents, “The DORA Declaration” (Cagan 2013) and “The Leiden Manifesto” (Hicks et al. 2015). Both documents urge caution in using bibliometrics, and the Leiden Manifesto offers a “distillation of best practice in metrics-based research assessment” (page 430). The first point is that “quantitative evaluation should support qualitative, expert assessment.”

Data Sources for Bibliometrics Research

Several APIs and portals are available for gathering bibliometric data. Each platform has its benefits and biases, and ideally multiple should be queried for any given project to ensure more complete data capture.

Resources

• For help using an API to collect bibliometric data, contact a Research Librarian.
• See blog post by DataLab postdoc Jane Carlen for using bibliometric data from Google Scholar to create a coauthor network in R.
• For help when there isn’t an API, drop in to DataLab office hours.

References

Content for this page was created in July 2019 by Dan Hicks, former DataLab postdoc and now Professor at UC Merced. This page is edited and maintained by DataLab staff.