A Citation-Based Approach to Detecting PaperMill Activity

Synopsis

This research-based study analyzes whether significant reference overlap among papers could be used as an early detection signal for papermill papers and additional research integrity issues. Using the citation data, the analysis shows that many papers share a substantial portion of their reference lists, often accompanied by structural similarities and irrelevant citations. The findings suggest that reference overlap can serve as an effective early indicator for detecting potentially problematic manuscripts. The study will be presented as an oral presentation.

Introduction

The rise of publications from papermills has become increasingly significant in recent years (Zimmer, 2025). Under pressure to publish, some researchers turn to such organization to produce low-quality manuscripts in exchange for financial compensation (Sabel & Larhammar, 2025). Papers produced by papermills are inherently low quality, mass produced and unethical (Sabel et al., 2025). Therefore, it is essential to screen manuscripts before peer review to reduce the workload for reviewers and prevent such low-quality papers from entering the published scientific record.

The papers produced by papermills often exhibit recurring templates, structural/layout similarities, recycled content, and multiple research integrity issues. Two particularly concerning issues are the reuse of reference lists from other published papers, and the inclusion of irrelevant citations in the papers. This study aims to analyze the references of papers from journals containing significant overlap of references.

Research questions

· Can reference overlap among paper serve as a quantitative indicator of reuse of paper templates by papermills?

· Is high reference similarity among papers an indicator of additional research integrity concerns? (e.g., irrelevant citations, template reuse, structural similarity, PubPeer reports)?

Methodology

In this study, we analyzed reference patterns of such suspected papermill papers from three journals. The selection of these journals was based on a preliminary analysis of a citation network. Specifically, the top three journals within a citation cluster exhibiting the highest number of retractions were selected for further analysis. The papers were obtained using the OpenAlex database. and the references of the papers were obtained using Opencitations API. No specific timeframe was imposed; all papers from the selected journals available in OpenAlex were included in the analysis. The reference overlap was measured using the Jaccard similarity metric, defined as the number of common references among two papers divided by the total number of unique references across both the papers.

Results

Our findings show that approximately 1045 ,149 and 102 papers in these journals, respectively, exhibit more than 30% overlap of references with at least one other paper published in the same journal, with some papers show even up to 100% reference overlap. Papers with high reference overlap frequently share similar structural layouts and integrity concerns reported on PubPeer (PubPeer - Search Publications and Join the Conversation., n.d.), despite being authored by different individuals. Paper sharing a significant number of references also often maintain the same reference order in their lists.

Conclusion

The most prevalent integrity issue among the papers was the inclusion of irrelevant citations i.e. the references papers are unrelated to the text where they are being referenced. This may serve to artificially boost citation counts for authors who are also customers of the same paper mill. Our results suggest that systematic screening for unusually high reference overlap between manuscripts and previously published articles could serve as an effective early filtering mechanism. The study would be of interests to academics, journal editors and reviewers. Such an approach may help editors and reviewers flag potentially problematic submissions and prompt deeper investigation into underlying research integrity concerns.

Future work of this study will include analyzing additional journals exhibiting similar patterns and automating the process to finding integrity issues in papers based on significant similar reference usage.

Keywords: Papermills, Citation manipulation, Academic integrity, Ethical publishing practices, Shared citations.

References:

[1] Zimmer, C. (2025, August 4). Fraudulent Scientific Papers Are Rapidly Increasing, Study Finds. The New York Times. https://www.nytimes.com/2025/08/04/science/04hs-science-papers-fraud-research-paper-mills.html

[2] Sabel, B., & Larhammar, D. (2025). Reformation of science publishing: the Stockholm Declaration. Royal Society Open Science, 12(11). https://doi.org/10.1098/rsos.251805

[3] Sabel, B. A., Knaack, E., Gerd Gigerenzer, & Mirela-Ioana Bilc. (2025). Fake publications in biomedical science: red-flagging method indicates mass production. Naunyn-Schmiedeberg S Archives of Pharmacology. https://doi.org/10.1007/s00210-025-04275-9

[4] PubPeer - Search publications and join the conversation. (n.d.). Pubpeer.com. https://pubpeer.com/