Identification of Peer Review Comments Types and Research on their Distribution at Different Citation Frequencies

Qin Chenglei; Han Ruxue; Zhou Haomin; Zhong Jiangtao; Zhang Chengzhi

doi:10.13266/j.issn.0252-3116.2022.13.010

Library and Information Service >

2022 , Vol. 66 >Issue 13: 102 - 117

DOI: https://doi.org/10.13266/j.issn.0252-3116.2022.13.010

INFORMATION RESEARCH

Identification of Peer Review Comments Types and Research on their Distribution at Different Citation Frequencies

Qin Chenglei ,
Han Ruxue ,
Zhou Haomin ,
Zhong Jiangtao ,
Zhang Chengzhi

Expand

Department of Information Management, Nanjing University of Science and Technology, Nanjing 210094

Received date: 2022-03-07

Revised date: 2022-05-11

Online published: 2022-07-06

Fold

Abstract

[Purpose/Significance] Identifying the types of peer review comments of academic articles and analyzing the distribution of the types of peer review comments in peer review reports at different paper citation frequencies will help to deepen the understanding of the peer review mechanism and provide new ideas for evaluating the academic quality of articles and quantifying the contribution of reviewers. [Method/Process] Firstly, the types of peer review comments were divided into six classifications: positive evaluation, negative evaluation, requirements/suggestions (primary and secondary aspects), problems/questions, and statements. After manual labeling, obtaining training and testing corpus, the performance of traditional machine learning models and deep learning models on the automatic identification of peer review comments types were compared and analyzed; Secondly, the academic papers covered by peer-reviewed reports were clustered thematically, and the citation frequencies were standardized. Finally, Spearman correlation coefficient, cumulative distribution, K-S test and negative binomial regression were used to analyze the distribution of peer review comments types in peer review reports at different citation frequencies. [Result/Conclusion] The SeiBert model is the best for recognizing peer review comments classifications. In Spearman correlation analysis, the distribution proportion of positive evaluation in the review reports has a significant weak positive correlation with the cited frequency. The distribution proportion of negative evaluation has a significant weak negative correlation with the cited frequency. Through the cumulative distribution, it is found that in most cases, when the cumulative probability is the same, the distribution proportion of positive evaluation in high cited areas is more significant than that in low cited areas, and the distribution proportion of Negative evaluation is less than that in low cited areas, and the K-S test can detect this difference. In the negative binomial regression analysis, the proportion of positive and negative evaluation distribution has significant positive and negative effects on the citations respectively. The result of this paper shows that the distribution of positive and negative evaluations in peer review reports is related to the cited frequencies of corresponding papers, which can reflect the academic quality of papers to a certain extent.

Key words： peer review; peer review comments; peer review comments types; cited frequencies; correlation analysis

Cite this article

Qin Chenglei , Han Ruxue , Zhou Haomin , Zhong Jiangtao , Zhang Chengzhi . Identification of Peer Review Comments Types and Research on their Distribution at Different Citation Frequencies[J]. Library and Information Service, 2022 , 66(13) : 102 -117 . DOI: 10.13266/j.issn.0252-3116.2022.13.010

References

[1] ELIZABETH M. Advancing peer review at BMC[EB/OL].[2022-02-10]. https://www.biomedcentral.com/about/advancing-peer-review.
[2] ULRICH P. Interactive journal concept for improved scientific publishing and quality assurance[J]. Learned publishing, 2004, 17(2):105-113.
[3] 秦成磊,章成志.大数据环境下同行评议面临的问题与对策[J].情报理论与实践, 2021, 44(4):99-112.
[4] ASAPBIO. Open letter on the publication of peer review reports[EB/OL].[2022-02-10]. https://asapbio.org/letter.
[5] NATURE COMMUNICATION EDITORIAL. Transparent peer review at nature communications[J]. Nature communications, 2015,6(10277):1-2.
[6] INGA V. Peer reviewers unmasked:largest global survey reveals trends[EB/OL].[2022-02-15]. https://www.nature.com/articles/d41586-018-06602-y.
[7] PUBLONS. Global state of peer review[R/OL].[2022-02-15]. https://publons.com/community/gspr.
[8] RITU D. Recognising the contribution of nature research journal referees[EB/OL].[2022-02-15]. http://blogs.nature.com/ofschemesandmemes/2019/04/16/recognising-the-contribution-of-nature-research-journal-referees.
[9] 林崇德.心理学大辞典[M].上海:上海教育出版社, 2003.
[10] ZHANG G Y, WANG L C, XIE W X, et al. "This article is interesting, however":exploring the language use in the peer review comment of articles published in the BMJ[J]. Aslib journal of information management, 2022, 74(3):399-416.
[11] WANG K, WAN X. Sentiment analysis of peer review texts for scholarly papers[C]//The 41st international acm sigir conference on research&development in information retrieval. New York:Association for Computing Machinery, 2018:175-184.
[12] GHOSAL T, VERMA R, EKBAL A, et al. DeepSentiPeer:harnessing sentiment in review texts to recommend peer review decisions[C]//Proceedings of the 57th annual meeting of the association for computational linguistics. Stroudsburg:Association for Computational Linguistics, 2019:1120-1130.
[13] BRAVO G, GRIMALDO F, LÓPEZ-I E, et al. The effect of publishing peer review reports on referee behavior in five scholarly journals[J]. Nature communication, 2019, 322(10):1-8.
[14] CHAKRABORTY S, GOYAL P, MUKHERJEE A. Aspect-based sentiment analysis of scientific reviews[C]//Proceedings of the ACM/IEEE joint conference on digital libraries in 2020. New York:Association for Computing Machinery, 2020:207-216.
[15] BULJAN I, GARCIA-COSTA D, GRIMALDO F, et al. Large-scale language analysis of peer review reports[J]. Elife sciences, 2020, 9:e53249.
[16] AUSLOOS M, NEDIC O, FRONCZAK A, et al. Quantifying the quality of peer reviewers through zipf's law[J]. Scientometrics, 2016, 106(1):347-368.
[17] RAMACHANDRAN L, GEHRINGER E, YADAV R. Automated assessment of the quality of peer reviews using natural language processing techniques[J]. International journal of artificial intelligence in education, 2017, 27:534-581.
[18] LI J, SATO A, SHIMURA K, et al. Multi-task peer-review score prediction[C]//Proceedings of the first workshop on scholarly document processing. Stroudsburg:Association for Computational Linguistics, 2020:121-126.
[19] 张琳, GUNNAR S.科学计量与同行评议相结合的科研评价——国际经验与启示[J].情报学报, 2020, 39(8):806-816.
[20] LI S, ZHAO W, YIN E, et al. A neural citation count prediction model based on peer review text[C]//Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing. Stroudsburg:Association for Computational Linguistics, 2019:4916-4926.
[21] ZONG Q, FAN L, XIE Y, et al. The relationship of polarity of post-publication peer review to citation count:evidence from publons[J]. Online information review, 2020,44(3):583-602.
[22] ZONG Q, XIE Y, LIANG J. Does open peer review improve citation count?evidence from a propensity score matching analysis of PeerJ[J]. Scientometrics, 2020, 125(1):607-623.
[23] HUA X, NIKOLOV M, BADUGU N, et al. Argument mining for understanding peer reviews[EB/OL].[2022-02-15]. https://arxiv.org/abs/1903.10104v1.
[24] FROMM M, FAERMAN E, BERRENDORF M, et al. Argument mining driven analysis of peer-reviews[EB/OL].[2022-02-15]. https://arxiv.org/abs/2012.07743.
[25] CHO K. Machine classification of peer comments in physics[C]//Proceedings of the 1st International conference on educational data mining. Montreal:International Educational Data Mining Society, 2008:192-196.
[26] LIAO C, LI Y, CHENG H. Examining the effects of automatic comment classification on comment types in peer review for graduate students'academic writing[C]//Proceedings of the 8th international congress on advanced applied informatics. Piscataway:IEEE Computer Society, 2019:215-220.
[27] GHOSAL T, KUMAR S, BHARTI P K, et al. Peer review analyze:a novel benchmark resource for computational analysis of peer reviews. Plos one, 2022, 17(1):e0259238.
[28] BHANDARI M, BUSSE J, DEVEREAUX P J, et al. Factors associated with citation rates in the orthopedic literature[J]. Canadian journal of surgery, 2007, 50(2):119-123.
[29] DIDEGAH F, BOWMAN T D, HOLMBERG K. On the differences between citations and altmetrics:an investigation of factors driving altmetrics versus citations for finnish articles[J]. Journal of the Association for Information Acience and Technology, 2018, 69(6):832-843.
[30] 王嘉鑫.科学合作与学术影响力相关分析[D].大连:大连理工大学, 2019.
[31] WEBSTER G D, JONASON P K, SCHEMBER T O, et al. Hot topics and popular papers in evolutionary psychology:analyses of title words and citation counts in evolution and human behavior[J]. Evolutionary psychology, 2009, 7(3):348-362.
[32] VIEIRA E S, GOMES J A. Citations to scientific articles:its distribution and dependence on the article features[J]. Journal of informetrics, 2010, 4(1):1-13.
[33] BOYACK K W, KLAVANS R. Predicting the importance of current papers[C]//Proceedings of the 10th international conference of the International Society for Scientometrics and Informetrics. Stockholm:Karolinska University Press, 2005, 1:335-342.
[34] LANCHO-BARRANTES B S, GUERRERO-BOTE V P, MOYA-ANEGON F. What lies behind the averages and significance of citation indicators in different disciplines?[J]. Journal of information science, 2010, 36(3):371-382.
[35] BORNMANN L, SCHIER H, MARX W, et al. What factors determine citation counts of publications in chemistry besides their quality?[J]. Journal of informetrics, 2012, 6(1):11-18.
[36] 姜磊,林德明.参考文献对论文被引频次的影响研究[J].科研管理, 2015, 36(1):121-126.
[37] 李力,刘德洪,张灿影.基于知识流动理论的科技论文学术影响力评价研究[J].情报科学, 2016, 34(7):113-119.
[38] 赵蓉英,郭凤娇,谭洁.基于Altmetrics的学术论文影响力评价研究——以汉语言文学学科为例[J].中国图书馆学报, 2016, 42(1):96-108.
[39] PETERS H P, VAN RAAN A F. On determinants of citation scores-a case study in chemical engineering[J]. Journal of the American Society for Information Science, 1994, 45(1):39-49.
[40] 牟象禹,龚凯乐,谢娟,等.论文被引频次的影响因素研究——以国内图书情报领域为例[J].图书情报知识, 2018(4):43-52.
[41] LEIMU R, KORICHEVA J. What determines the citation frequency of ecological papers?[J]. Trends in ecology&evolution, 2005, 20(1):28-32.
[42] VANCLAY J K. Factors affecting citation rates in environmental science[J]. Journal of informetrics, 2013, 7(2):265-271.
[43] NI J, ZHAO Z, SHAO Y, et al. The influence of opening peer review on the citations of journal articles[J]. Scientometrics, 2021, 126(12):9393-9404.
[44] BORNMANN L. The problem of citation impact assessments for recent publication years in institutional evaluations[J]. Journal of informetrics, 2013, 7(3):722-729.
[45] CHAWLA D. In brief, papers with shorter titles get more citations, study suggests[EB/OL].[2021-10-30]. https://www.science.org/content/article/brief-papers-shorter-titles-get-more-citations-study-suggests.
[46] HASLAM N, BAN L, KAUFMANN L, et al. What makes an article influential?Predicting impact in social and personality psychology[J]. Scientometrics, 2008, 76(1):169-185.
[47] CHEN C. Predictive effects of structural variation on citation counts[J]. Journal of the American Society for Information Science, 2012, 63(3):431-449.
[48] ZHAO D Z. Characteristics and impact of grant-funded research:a case study of the library and information science field[J]. Scientometrics, 2010, 84(2):293-306.
[49] ROSHANI S, BAGHERYLOOIEH M R, MOSLEH M, et al. What is the relationship between research funding and citation-based performance?a comparative analysis between critical disciplines[J]. Scientometrics, 2021, 126(9):7859-7874.
[50] ANNALINGAM A, DAMAYANTHI H, JAYAWARDENA R, et al. Determinants of the citation rate of medical research publications from a developing country[J]. Springerplus, 2014, 3(1):1-6.
[51] STREMERSCH S, CAMACHO N, VANNESTE S, et al. Unraveling scientific impact:citation types in marketing journals[J]. International journal of research in marketing, 2015, 32(1):64-77.
[52] FU L D, ALIFERIS C F. Using content-based and bibliometric features for machine learning models to predict citation counts in the biomedical literature[J]. Scientometrics, 2010, 85(1):257-270.
[53] ZHANG X Y, XIE Q, SONG M. Measuring the impact of novelty, bibliometric, and academic-network factors on citation count using a neural network[J]. Journal of informetrics, 2021, 15(2):1-17.
[54] MA C, LI Y, GUO F, et al. The citation trap:papers published at year-end receive systematically fewer citations[J]. Journal of economic behavior&organization, 2019, 166:667-687.
[55] 伍军红,肖宏,任美亚,等. PCSI:一种单篇论文被引频次标准化方法[J].图书情报工作,2020,64(23):22-30.
[56] DEVLIN J, CHANG M, LEE K, et al. BERT:pre-training of deep bidirectional transformers for language understanding[EB/OL].[2021-07-15]. https://arxiv.org/abs/1810.04805.
[57] BELTAGY Z, LO K, COHAN A. SciBert:pretrained language model for scientific text[EB/OL].[2021-07-15]. https://arxiv.org/abs/1903.10676.
[58] QIN C, ZHANG C, BU Y. Exploring the distribution regularities of user attention and sentiment toward product aspects in online reviews[J]. The electronic library, 2021, 39(4):615-638.
[59] 秦成磊,章成志.中文在线评论中的产品新属性识别研究[J].信息资源管理学报, 2020, 10(3):78-91.
[60] LE Q, MIKOLOV T. Distributed representations of sentences and documents[EB/OL].[2021-07-15]. https://arxiv.org/pdf/1405.4053v2.
[61] 周志华.机器学习[M].北京:清华大学出版社, 2016.
[62] PARK E, KIM W. What are the drivers of citations?application in tourism and hospitality journals[J]. Applied sciences, 2021, 11(19):1-16.
[63] DAUD A, AMJAD T, SIDDIQUI M A, et al. Correlational analysis of topic specificity and citations count of publication venues[J]. Library hi tech, 2019, 37(1):8-18.
[64] KOCH L. An application of hierarchical kappa-type statistics in the assessment of majority agreement among multiple observers[J]. Biometrics, 1977, 33(2):363-374.
[65] CHAWLA N, BOWYER K, HALL L, et al. Smote:synthetic minority over-sampling technique[J]. Journal of artificial intelligence research, 2002, 16(1):321-357.
[66] QU Y, SHEN D, SHEN Y, et al. Coda:contrast-enhanced and diversity-promoting data augmentation for natural language understanding[EB/OL].[2021-07-15]. https://arxiv.org/abs/2010.08670.
[67] 宗成庆,夏睿,张家俊.文本数据挖掘[M].北京:清华大学出版社, 2019.
[68] CORTES C, LAWRENCE N D. Inconsistency in conference peer review:revisiting the 2014 neurips experiment[EB/OL].[2021-10-30]. https://arxiv.org/abs/2109.09774.
[69] SILER K, LEE K, BERO L. Measuring the effectiveness of scientific gatekeeping[J]. Proceedings of the National Academy of Sciences, 2015, 112(2):360-365.作者贡献说明:秦成磊:提出研究思路与研究设计、数据采集与处理、数据标注、进行实验、论文撰写、论文修订;韩茹雪:数据标注;周昊旻:数据标注;仲江涛:数据标注;章成志:研究设计与研究思路讨论,论文修订。

Options

Outlines

模态框（Modal）标题

Abstract

Cite this article

References