[Purpose/Significance] Peer review serves as a pivotal quality assurance mechanism in academic knowledge production, encompassing implicit semantic insights within reviewers’ comments. Using text classification models to identify and analyze the semantic features in comment texts can deepen our understanding of the peer review process, illuminate the opaque processes of peer review, and ultimately enhance this crucial system. [Method/Process] This paper introduced an “aspect-intention” semantic mining framework to delineate the specific aspects of papers that reviewers scrutinize and the intentions underpinning their comments. Leveraging data from the open peer review journal eLife, it constructed an annotated dataset DAIPRV1. It then trained and tested text multi-classification models based on SciBERT to identify the aspects and intentions of reviewers’ comments. Finally, it analyzed the semantic features based on the classification results. [Result/Conclusion] The analysis reveals that reviewers predominantly focus on the empirical aspects of papers, followed by presentational and theoretical elements. In terms of intention, reviewers commonly give instructions, urging authors to clarify their work, provide additional experiments or evidence, and make editing and formatting revisions. This is followed by evaluative and summarizing comments, with multiple instructions integrated within a single comment. This paper also underscores the correspondence relationship between the aspects of reviewers’ comments and their underlying intentions. By exploring the semantic features from the perspective of “aspect-intention” correlation, this paper provides a rich perspective for understanding the peer review process, and deepens the comprehension of peer review as an important form of academic communication.
[1] MERTON R K. The sociology of science: theoretical and empirical investigations[M]. Chicago: University of Chicago Press, 1973: 339.
[2] 楚宾, 哈克特. 难有同行的科学：同行评议与美国科学政策[M]. 北京: 北京大学出版社, 2011. (CHUBIN D E, HACKETT E J. Peerless science: peer review and U.S. science policy[M]. Beijing: Peking University Press, 2011.)
[3] MULLIGAN A, HALL L, RAPHAEL E. Peer review in a changing world: an international study measuring the attitudes of researchers[J]. Journal of the American Society for Information Science and Technology, 2013, 64(1): 132-161.
[4] CARNEIRO C F D, Queiroz V G S, Moulin T C, et al. Comparing quality of reporting between preprints and peer-reviewed articles in the biomedical literature[J]. Research integrity and peer review, 2020, 5(1): 16.
[5] BENOS D J, BASHARI E, CHAVES J M, et al. The ups and downs of peer review[J]. Advances in physiology education, 2007, 31(2): 145-152.
[6] THURNER S, HANEL R. Peer-review in a world with rational scientists: toward selection of the average[J]. The European physical journal B, 2011, 84(4): 707-711.
[7] ESAREY J. Does peer review identify the best papers? a simulation study of editors, reviewers, and the scientific publication process[J]. Political science & politics, 2017, 50(4): 963-969.
[8] COWLEY S J. How peer-review constrains cognition: on the frontline in the knowledge sector[J]. Frontiers in psychology, 2015, 6:1706.
[9] WOOD M. Beyond journals and peer review: towards a more flexible ecosystem for scholarly communication[M/OL]. [2024-12-05]. https://www.qeios.com/read/SWKKOC.
[10] TEPLITSKIY M. Frame search and re-search: how quantitative sociological articles change during peer review[J]. The American sociologist, 2016, 47(2): 264-288.
[11] 孙梦婷, 张琳. 从“要素公开”到“开放参与”：开放同行评议策略研究[J]. 图书情报知识, 2024, 41(2): 67-80. (SUN M, ZHANG L. From “element transparency” to “open participation”: an investigation on open peer review strategies[J]. Documentation, information & knowledge, 2024, 41(2): 67-80.)
[12] FALK DELGADO A, GARRETSON G, FALK DELGADO A. The language of peer review reports on articles published in the BMJ, 2014–2017: an observational study[J]. Scientometrics, 2019, 120(3): 1225-1235.
[13] SUN Z, CLARK CAO C, MA C, et al. The academic status of reviewers predicts their language use[J]. Journal of informetrics, 2023, 17(4): 101449.
[14] BULJAN I, GARCIA-COSTA D, GRIMALDO F, et al. Large-scale language analysis of peer review reports[J]. eLife, 2020, 9.
[15] STEPHEN D. Peer reviewers equally critique theory, method, and writing, with limited effect on the final content of accepted manuscripts[J]. Scientometrics, 2022, 127(6): 3413-3435.
[16] MUNGRA P, WEBBER P. Peer review process in medical research publications: language and content comments[J]. English for specific purposes, 2010, 29(1): 43-53.
[17] HERBER O R, BRADBURY-JONES C, BÖLING S, et al. What feedback do reviewers give when reviewing qualitative manuscripts? A focused mapping review and synthesis[J]. BMC medical research methodology, 2020, 20(1): 122.
[18] QIN C, ZHANG C. Which structure of academic articles do referees pay more attention to?: perspective of peer review and full-text of academic articles[J]. Aslib journal of information management, 2023, 75(5): 884-916.
[19] ZONG Q, XIE Y, LIANG J. Does open peer review improve citation count? evidence from a propensity score matching analysis of PeerJ[J]. Scientometrics, 2020, 125(1): 607-623.
[20] CHAKRABORTY S, GOYAL P, MUKHERJEE A. Aspect-based sentiment analysis of scientific reviews[C] //Proceedings of the ACM/IEEE joint conference on digital libraries in 2020. Wuhan: Association for Computing Machinery, 2020: 207-216.
[21] GHOSAL T, KUMAR S, BHARTI P K, et al. Peer review analyze: a novel benchmark resource for computational analysis of peer reviews[J]. Plos one, 2022, 17(1): e0259238.
[22] YUAN W, LIU P, NEUBIG G. Can we automate scientific reviewing?[J]. Journal of articial intelligence research, 2022, 75: 42.
[23] CHENG L, BING L, YU Q, et al. APE: argument pair extraction from peer review and rebuttal via multi-task learning[C]//Proceedings of the 2020 conference on empirical methods in natural language processing. Association for Computational Linguistics, 2020: 7000-7011.
[24] FROMM M, FAERMAN E, BERRENDORF M, et al. Argument mining driven analysis of peer-reviews[C]//Proceedings of the AAAI conference on artificial intelligence, 2021: 4758-4766.
[25] HUA X, NIKOLOV M, BADUGU N, et al. Argument mining for understanding peer reviews[C]//Minneapolis: Association for Computational Linguistics, 2019: 2131-2137.
[26] KENNARD N, O’GORMAN T, DAS R, et al. DISAPERE: a dataset for discourse structure in peer review discussions[A]. Seattle, United States: Association for Computational Linguistics. 2022: 1234-1249.
[27] KING S R F. Consultative review is worth the wait[J]. eLife, 2017, 6: e32012.
[28] 刘丽萍, 刘春丽. eLife开放同行评审模式研究[J]. 中国科技期刊研究, 2019, 30(9): 949-955. (LIU L, LIU C. Research on the open peer review mode of eLife[J]. Chinese journal of scientific and technical periodicals, 2019, 30(9): 949-955.)
[29] 于曦. eLife开放同行评议模式改革与启示[J]. 中国科技期刊研究, 2023, 34(5): 609-614. (YU X. eLife’s open peer review model reform and enlightenment[J]. Chinese journal of scientific and technical periodicals, 2023, 34(5): 609-614.)
[30] BORDAGE G. Reasons reviewers reject and accept manuscripts: the strengths and weaknesses in medical education reports[J]. Academic medicine, 2001, 76(9): 889-896.
[31] STRANG D, SILER K. Revising as reframing: original submissions versus published papers in Administrative Science Quarterly, 2005 to 2009[J]. Sociological theory, 2015, 33(1): 71-96.
[32] KANG D, AMMAR W, DALVI B, et al. A dataset of peer reviews (PeerRead): collection, insights and NLP applications[A]. New Orleans, Louisiana: Association for Computational Linguistics. 2018: 1647-1661.
[33] KUZNETSOV I, BUCHMANN J, EICHLER M, et al. Revise and resubmit: an intertextual model of text-based collaboration in peer review[J]. Computational linguistics, 2022, 48(4): 949-986.
[34] 秦成磊, 韩茹雪, 周昊旻，等. 同行评审意见类型识别及其在不同被引频次下的分布研究[J]. 图书情报工作, 2022, 66(13): 102-117. (QIN C, HAN R, ZHOU H, et al. Identification of peer review comments types and research on their distribution at different citation frequencies[J]. Library and information service, 2022, 66(13): 102-117.)
[35] WANG K, WAN X. Sentiment analysis of peer review texts for scholarly papers[C]//The 41st international ACM SIGIR conference on research & development in information retrieval. New York: ACM, 2018.
[36] BHARTI P K, GHOSAL T, AGARWAL M, et al. BetterPR: a dataset for estimating the constructiveness of peer review comments[M]. Cham: Springer, 2022: 500-505.
