图书情报工作

图书情报工作 >

2022 , Vol. 66 >Issue 13: 102 - 117

DOI: https://doi.org/10.13266/j.issn.0252-3116.2022.13.010

情报研究

同行评审意见类型识别及其在不同被引频次下的分布研究

  • 秦成磊 ,
  • 韩茹雪 ,
  • 周昊旻 ,
  • 仲江涛 ,
  • 章成志
展开
  • 南京理工大学信息管理系 南京 210094
秦成磊,博士研究生;韩茹雪,本科生;周昊旻,本科生;仲江涛,本科生。

收稿日期: 2022-03-07

  修回日期: 2022-05-11

  网络出版日期: 2022-07-06

收起

Identification of Peer Review Comments Types and Research on their Distribution at Different Citation Frequencies

  • Qin Chenglei ,
  • Han Ruxue ,
  • Zhou Haomin ,
  • Zhong Jiangtao ,
  • Zhang Chengzhi
Expand
  • Department of Information Management, Nanjing University of Science and Technology, Nanjing 210094

Received date: 2022-03-07

  Revised date: 2022-05-11

  Online published: 2022-07-06

Fold

摘要

[目的/意义]识别学术论文同行评审意见类型、分析不同被引频次下同行评审意见类型在同行评审报告中的分布情况,有助于加深对同行评议机制的认识,为评估论文学术质量、量化评审专家贡献提供新思路。[方法/过程]首先,将同行评审意见类型划分为正面评价、负面评价、要求/建议(主、次要方面)、问题/疑问、陈述六个类别,经人工标注、获取训练、测试语料后,对比分析传统机器学习模型、深度学习模型在同行评审意见类型自动识别上的效果;其次,将同行评审报告涉及的学术论文进行主题聚类,进而对被引频次进行标准化处理;最后,使用Spearman相关系数、累积分布、K-S检验、负二项回归分析不同被引频次学术论文对应的同行评审报告中同行评审意见类型的分布情况。[结果/结论]SciBert模型识别效果最佳;在基于Spearman的相关性分析中,评审报告中正面评价的分布占比与被引频次具有显著的弱正相关,负面评价的分布占比与被引频次具有显著的弱负相关;通过累计分布发现,多数情况下,当累积概率相同时,高被引分区中正面评价的分布占比大于低被引分区、负面评价的分布占比小于低被引分区,K-S检验能够检测到这种差异;在负二项回归分析中,正面评价分布占比、负面评价分布占比分别对被引频次有显著的正向影响、负向影响。研究结果表明,同行评审报告中正面评价、负面评价的分布情况与其对应论文的被引频次存在相关性,被引频次一定程度上能够反映论文的学术质量。

关键词： 同行评议; 同行评审意见; 同行评审意见类型; 被引频次; 相关性分析

本文引用格式

秦成磊 , 韩茹雪 , 周昊旻 , 仲江涛 , 章成志 . 同行评审意见类型识别及其在不同被引频次下的分布研究[J]. 图书情报工作, 2022 , 66(13) : 102 -117 . DOI: 10.13266/j.issn.0252-3116.2022.13.010

Abstract

[Purpose/Significance] Identifying the types of peer review comments of academic articles and analyzing the distribution of the types of peer review comments in peer review reports at different paper citation frequencies will help to deepen the understanding of the peer review mechanism and provide new ideas for evaluating the academic quality of articles and quantifying the contribution of reviewers. [Method/Process] Firstly, the types of peer review comments were divided into six classifications: positive evaluation, negative evaluation, requirements/suggestions (primary and secondary aspects), problems/questions, and statements. After manual labeling, obtaining training and testing corpus, the performance of traditional machine learning models and deep learning models on the automatic identification of peer review comments types were compared and analyzed; Secondly, the academic papers covered by peer-reviewed reports were clustered thematically, and the citation frequencies were standardized. Finally, Spearman correlation coefficient, cumulative distribution, K-S test and negative binomial regression were used to analyze the distribution of peer review comments types in peer review reports at different citation frequencies. [Result/Conclusion] The SeiBert model is the best for recognizing peer review comments classifications. In Spearman correlation analysis, the distribution proportion of positive evaluation in the review reports has a significant weak positive correlation with the cited frequency. The distribution proportion of negative evaluation has a significant weak negative correlation with the cited frequency. Through the cumulative distribution, it is found that in most cases, when the cumulative probability is the same, the distribution proportion of positive evaluation in high cited areas is more significant than that in low cited areas, and the distribution proportion of Negative evaluation is less than that in low cited areas, and the K-S test can detect this difference. In the negative binomial regression analysis, the proportion of positive and negative evaluation distribution has significant positive and negative effects on the citations respectively. The result of this paper shows that the distribution of positive and negative evaluations in peer review reports is related to the cited frequencies of corresponding papers, which can reflect the academic quality of papers to a certain extent.

Key words： peer review; peer review comments; peer review comments types; cited frequencies; correlation analysis

参考文献

[1] ELIZABETH M. Advancing peer review at BMC[EB/OL].[2022-02-10]. https://www.biomedcentral.com/about/advancing-peer-review.
[2] ULRICH P. Interactive journal concept for improved scientific publishing and quality assurance[J]. Learned publishing, 2004, 17(2):105-113.
[3] 秦成磊,章成志.大数据环境下同行评议面临的问题与对策[J].情报理论与实践, 2021, 44(4):99-112.
[4] ASAPBIO. Open letter on the publication of peer review reports[EB/OL].[2022-02-10]. https://asapbio.org/letter.
[5] NATURE COMMUNICATION EDITORIAL. Transparent peer review at nature communications[J]. Nature communications, 2015,6(10277):1-2.
[6] INGA V. Peer reviewers unmasked:largest global survey reveals trends[EB/OL].[2022-02-15]. https://www.nature.com/articles/d41586-018-06602-y.
[7] PUBLONS. Global state of peer review[R/OL].[2022-02-15]. https://publons.com/community/gspr.
[8] RITU D. Recognising the contribution of nature research journal referees[EB/OL].[2022-02-15]. http://blogs.nature.com/ofschemesandmemes/2019/04/16/recognising-the-contribution-of-nature-research-journal-referees.
[9] 林崇德.心理学大辞典[M].上海:上海教育出版社, 2003.
[10] ZHANG G Y, WANG L C, XIE W X, et al. "This article is interesting, however":exploring the language use in the peer review comment of articles published in the BMJ[J]. Aslib journal of information management, 2022, 74(3):399-416.
[11] WANG K, WAN X. Sentiment analysis of peer review texts for scholarly papers[C]//The 41st international acm sigir conference on research&development in information retrieval. New York:Association for Computing Machinery, 2018:175-184.
[12] GHOSAL T, VERMA R, EKBAL A, et al. DeepSentiPeer:harnessing sentiment in review texts to recommend peer review decisions[C]//Proceedings of the 57th annual meeting of the association for computational linguistics. Stroudsburg:Association for Computational Linguistics, 2019:1120-1130.
[13] BRAVO G, GRIMALDO F, LÓPEZ-I E, et al. The effect of publishing peer review reports on referee behavior in five scholarly journals[J]. Nature communication, 2019, 322(10):1-8.
[14] CHAKRABORTY S, GOYAL P, MUKHERJEE A. Aspect-based sentiment analysis of scientific reviews[C]//Proceedings of the ACM/IEEE joint conference on digital libraries in 2020. New York:Association for Computing Machinery, 2020:207-216.
[15] BULJAN I, GARCIA-COSTA D, GRIMALDO F, et al. Large-scale language analysis of peer review reports[J]. Elife sciences, 2020, 9:e53249.
[16] AUSLOOS M, NEDIC O, FRONCZAK A, et al. Quantifying the quality of peer reviewers through zipf's law[J]. Scientometrics, 2016, 106(1):347-368.
[17] RAMACHANDRAN L, GEHRINGER E, YADAV R. Automated assessment of the quality of peer reviews using natural language processing techniques[J]. International journal of artificial intelligence in education, 2017, 27:534-581.
[18] LI J, SATO A, SHIMURA K, et al. Multi-task peer-review score prediction[C]//Proceedings of the first workshop on scholarly document processing. Stroudsburg:Association for Computational Linguistics, 2020:121-126.
[19] 张琳, GUNNAR S.科学计量与同行评议相结合的科研评价——国际经验与启示[J].情报学报, 2020, 39(8):806-816.
[20] LI S, ZHAO W, YIN E, et al. A neural citation count prediction model based on peer review text[C]//Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing. Stroudsburg:Association for Computational Linguistics, 2019:4916-4926.
[21] ZONG Q, FAN L, XIE Y, et al. The relationship of polarity of post-publication peer review to citation count:evidence from publons[J]. Online information review, 2020,44(3):583-602.
[22] ZONG Q, XIE Y, LIANG J. Does open peer review improve citation count?evidence from a propensity score matching analysis of PeerJ[J]. Scientometrics, 2020, 125(1):607-623.
[23] HUA X, NIKOLOV M, BADUGU N, et al. Argument mining for understanding peer reviews[EB/OL].[2022-02-15]. https://arxiv.org/abs/1903.10104v1.
[24] FROMM M, FAERMAN E, BERRENDORF M, et al. Argument mining driven analysis of peer-reviews[EB/OL].[2022-02-15]. https://arxiv.org/abs/2012.07743.
[25] CHO K. Machine classification of peer comments in physics[C]//Proceedings of the 1st International conference on educational data mining. Montreal:International Educational Data Mining Society, 2008:192-196.
[26] LIAO C, LI Y, CHENG H. Examining the effects of automatic comment classification on comment types in peer review for graduate students'academic writing[C]//Proceedings of the 8th international congress on advanced applied informatics. Piscataway:IEEE Computer Society, 2019:215-220.
[27] GHOSAL T, KUMAR S, BHARTI P K, et al. Peer review analyze:a novel benchmark resource for computational analysis of peer reviews. Plos one, 2022, 17(1):e0259238.
[28] BHANDARI M, BUSSE J, DEVEREAUX P J, et al. Factors associated with citation rates in the orthopedic literature[J]. Canadian journal of surgery, 2007, 50(2):119-123.
[29] DIDEGAH F, BOWMAN T D, HOLMBERG K. On the differences between citations and altmetrics:an investigation of factors driving altmetrics versus citations for finnish articles[J]. Journal of the Association for Information Acience and Technology, 2018, 69(6):832-843.
[30] 王嘉鑫.科学合作与学术影响力相关分析[D].大连:大连理工大学, 2019.
[31] WEBSTER G D, JONASON P K, SCHEMBER T O, et al. Hot topics and popular papers in evolutionary psychology:analyses of title words and citation counts in evolution and human behavior[J]. Evolutionary psychology, 2009, 7(3):348-362.
[32] VIEIRA E S, GOMES J A. Citations to scientific articles:its distribution and dependence on the article features[J]. Journal of informetrics, 2010, 4(1):1-13.
[33] BOYACK K W, KLAVANS R. Predicting the importance of current papers[C]//Proceedings of the 10th international conference of the International Society for Scientometrics and Informetrics. Stockholm:Karolinska University Press, 2005, 1:335-342.
[34] LANCHO-BARRANTES B S, GUERRERO-BOTE V P, MOYA-ANEGON F. What lies behind the averages and significance of citation indicators in different disciplines?[J]. Journal of information science, 2010, 36(3):371-382.
[35] BORNMANN L, SCHIER H, MARX W, et al. What factors determine citation counts of publications in chemistry besides their quality?[J]. Journal of informetrics, 2012, 6(1):11-18.
[36] 姜磊,林德明.参考文献对论文被引频次的影响研究[J].科研管理, 2015, 36(1):121-126.
[37] 李力,刘德洪,张灿影.基于知识流动理论的科技论文学术影响力评价研究[J].情报科学, 2016, 34(7):113-119.
[38] 赵蓉英,郭凤娇,谭洁.基于Altmetrics的学术论文影响力评价研究——以汉语言文学学科为例[J].中国图书馆学报, 2016, 42(1):96-108.
[39] PETERS H P, VAN RAAN A F. On determinants of citation scores-a case study in chemical engineering[J]. Journal of the American Society for Information Science, 1994, 45(1):39-49.
[40] 牟象禹,龚凯乐,谢娟,等.论文被引频次的影响因素研究——以国内图书情报领域为例[J].图书情报知识, 2018(4):43-52.
[41] LEIMU R, KORICHEVA J. What determines the citation frequency of ecological papers?[J]. Trends in ecology&evolution, 2005, 20(1):28-32.
[42] VANCLAY J K. Factors affecting citation rates in environmental science[J]. Journal of informetrics, 2013, 7(2):265-271.
[43] NI J, ZHAO Z, SHAO Y, et al. The influence of opening peer review on the citations of journal articles[J]. Scientometrics, 2021, 126(12):9393-9404.
[44] BORNMANN L. The problem of citation impact assessments for recent publication years in institutional evaluations[J]. Journal of informetrics, 2013, 7(3):722-729.
[45] CHAWLA D. In brief, papers with shorter titles get more citations, study suggests[EB/OL].[2021-10-30]. https://www.science.org/content/article/brief-papers-shorter-titles-get-more-citations-study-suggests.
[46] HASLAM N, BAN L, KAUFMANN L, et al. What makes an article influential?Predicting impact in social and personality psychology[J]. Scientometrics, 2008, 76(1):169-185.
[47] CHEN C. Predictive effects of structural variation on citation counts[J]. Journal of the American Society for Information Science, 2012, 63(3):431-449.
[48] ZHAO D Z. Characteristics and impact of grant-funded research:a case study of the library and information science field[J]. Scientometrics, 2010, 84(2):293-306.
[49] ROSHANI S, BAGHERYLOOIEH M R, MOSLEH M, et al. What is the relationship between research funding and citation-based performance?a comparative analysis between critical disciplines[J]. Scientometrics, 2021, 126(9):7859-7874.
[50] ANNALINGAM A, DAMAYANTHI H, JAYAWARDENA R, et al. Determinants of the citation rate of medical research publications from a developing country[J]. Springerplus, 2014, 3(1):1-6.
[51] STREMERSCH S, CAMACHO N, VANNESTE S, et al. Unraveling scientific impact:citation types in marketing journals[J]. International journal of research in marketing, 2015, 32(1):64-77.
[52] FU L D, ALIFERIS C F. Using content-based and bibliometric features for machine learning models to predict citation counts in the biomedical literature[J]. Scientometrics, 2010, 85(1):257-270.
[53] ZHANG X Y, XIE Q, SONG M. Measuring the impact of novelty, bibliometric, and academic-network factors on citation count using a neural network[J]. Journal of informetrics, 2021, 15(2):1-17.
[54] MA C, LI Y, GUO F, et al. The citation trap:papers published at year-end receive systematically fewer citations[J]. Journal of economic behavior&organization, 2019, 166:667-687.
[55] 伍军红,肖宏,任美亚,等. PCSI:一种单篇论文被引频次标准化方法[J].图书情报工作,2020,64(23):22-30.
[56] DEVLIN J, CHANG M, LEE K, et al. BERT:pre-training of deep bidirectional transformers for language understanding[EB/OL].[2021-07-15]. https://arxiv.org/abs/1810.04805.
[57] BELTAGY Z, LO K, COHAN A. SciBert:pretrained language model for scientific text[EB/OL].[2021-07-15]. https://arxiv.org/abs/1903.10676.
[58] QIN C, ZHANG C, BU Y. Exploring the distribution regularities of user attention and sentiment toward product aspects in online reviews[J]. The electronic library, 2021, 39(4):615-638.
[59] 秦成磊,章成志.中文在线评论中的产品新属性识别研究[J].信息资源管理学报, 2020, 10(3):78-91.
[60] LE Q, MIKOLOV T. Distributed representations of sentences and documents[EB/OL].[2021-07-15]. https://arxiv.org/pdf/1405.4053v2.
[61] 周志华.机器学习[M].北京:清华大学出版社, 2016.
[62] PARK E, KIM W. What are the drivers of citations?application in tourism and hospitality journals[J]. Applied sciences, 2021, 11(19):1-16.
[63] DAUD A, AMJAD T, SIDDIQUI M A, et al. Correlational analysis of topic specificity and citations count of publication venues[J]. Library hi tech, 2019, 37(1):8-18.
[64] KOCH L. An application of hierarchical kappa-type statistics in the assessment of majority agreement among multiple observers[J]. Biometrics, 1977, 33(2):363-374.
[65] CHAWLA N, BOWYER K, HALL L, et al. Smote:synthetic minority over-sampling technique[J]. Journal of artificial intelligence research, 2002, 16(1):321-357.
[66] QU Y, SHEN D, SHEN Y, et al. Coda:contrast-enhanced and diversity-promoting data augmentation for natural language understanding[EB/OL].[2021-07-15]. https://arxiv.org/abs/2010.08670.
[67] 宗成庆,夏睿,张家俊.文本数据挖掘[M].北京:清华大学出版社, 2019.
[68] CORTES C, LAWRENCE N D. Inconsistency in conference peer review:revisiting the 2014 neurips experiment[EB/OL].[2021-10-30]. https://arxiv.org/abs/2109.09774.
[69] SILER K, LEE K, BERO L. Measuring the effectiveness of scientific gatekeeping[J]. Proceedings of the National Academy of Sciences, 2015, 112(2):360-365.作者贡献说明:秦成磊:提出研究思路与研究设计、数据采集与处理、数据标注、进行实验、论文撰写、论文修订;韩茹雪:数据标注;周昊旻:数据标注;仲江涛:数据标注;章成志:研究设计与研究思路讨论,论文修订。
Options
文章导航

/

〈 〉