图书情报工作

图书情报工作 >

2024 , Vol. 68 >Issue 24: 124 - 132

DOI: https://doi.org/10.13266/j.issn.0252-3116.2024.24.011

情报研究

不同评审模式对出版偏倚的影响——基于文本挖掘的发现

  • 贺颖 ,
  • 刘小玲 ,
  • 王治钧
展开
  • 1 天津师范大学管理学院, 天津, 300387;
    2 中国林业科学研究院资源信息研究所, 北京 100091
贺颖，教授，博士，硕士生导师，E-mail:heying@tjnu.edu.cn；刘小玲，硕士研究生；王治钧，硕士。

收稿日期: 2024-03-07

  修回日期: 2024-06-14

  网络出版日期: 2024-12-23

基金资助

本文系国家社会科学基金一般项目“基于区块链的科学论文开放式同行评议质量控制研究” (项目编号：20BTQ084) 和天津师范大学研究生科研创新项目“基于注册报告模式的出版偏倚与出版质量的实证研究”（项目编号：2024KYCX080Y）研究成果之一。

收起

Impact of Different Review Modes on Publication Bias: Based on Text Mining

  • He Ying ,
  • Liu Xiaoling ,
  • Wang Zhijun
Expand
  • 1 School of Management, Tianjin Normal University, Tianjin 300387;
    2 Research Institute of Forest Resource Information Techniques, Chinese Academy of Forestry, Beijing 100091

Received date: 2024-03-07

  Revised date: 2024-06-14

  Online published: 2024-12-23

Supported by

This work is supported by the National Social Science Fund of China project titled “Research on Quality Control of Open Peer Review for Scientific Papers Based on Blockchain” (Grant No. 20BTQ084) and Postgraduate Research Innovation of Tianjin Normal University project titled “An Empirical Study on Publication Bias and Publication Quality Based on Registration Report Mode” (Grant No. 2024KYCX080Y).

Fold

摘要

[目的/意义] 通过对公开的同行评审报告进行文本挖掘，探寻隐藏在同行评审过程中的出版偏倚，为其解决方案提供理论支持。[方法/过程] 对透明式同行评审TPR和结果盲态评审RBR两种评审模式的公开评审报告进行文本挖掘，采用LDA主题分析法、情感分析法、内容分析法进行对比分析，观察评审人的关注点、情感倾向、研究结果极性分布差异。[结果/结论] 采用结果盲态评审RBR模式的评审人更注重研究方法与过程的质量与严谨性；而采用透明式同行评审TPR模式的评审人更关注学术论文的研究结果与研究结论，评审时情感更具主观性，在评审过程中容易导致“唯结果论”的出版偏倚现象。据此得到以下结论：论文评审环节的出版偏倚被证实；科研评价体系是造成出版偏倚现象的核心原因；积极推行评审过程的全要素开放模式；学界应转变出版观念、推行新兴出版模式。

关键词： 出版偏倚; 审稿专家; 同行评审

本文引用格式

贺颖 , 刘小玲 , 王治钧 . 不同评审模式对出版偏倚的影响——基于文本挖掘的发现[J]. 图书情报工作, 2024 , 68(24) : 124 -132 . DOI: 10.13266/j.issn.0252-3116.2024.24.011

Abstract

[Purpose/Significance] By conducting text mining on open peer review reports, this study aims to explore the hidden publication bias in the peer review mode and provide theoretical support for its solutions. [Method/Process] The text mining analysis was conducted on the review reports of two modes: Transparent Peer Review (TPR) and Results-Blind Review (RBR). LDA topic analysis, sentiment analysis, and content analysis were used to compare and observe the differences in reviewers’ focal points, emotional tendencies, and the distribution of research results polarity. [Result/Conclusion] Reviewers using the Results-Blind Review (RBR) model tend to focus more on the quality and rigor of research methods and processes, while those using the Transparent Peer Review (TPR) model pay more attention to the results and conclusions of academic papers, and the corresponding evaluations during the review process are more subjective, leading to a potential “results-oriented” publication bias. It draws the following conclusions. Firstly, publication bias in the paper review stage has been confirmed. Second, the core reason behind this bias is the research evaluation system. Third, actively promoting a fully open review process is necessary. Finally, the academic community should shift publishing perspectives and promote emerging publishing models.

Key words： publication bias; reviewer; peer review

参考文献

[1] FANELLI D. Negative results are disappearing from most disciplines and countries[J]. Scientometrics, 2012, 90(3): 891-904.
[2] STEFAN A, SCHÖNBRODT F. Big little lies: a compendium and simulation of p-hacking strategies[J]. Royal Society open science, 2023, 10(2): 220346.
[3] MURPHY K, AGUINIS H. Harking: how badly can cherry-picking and question trolling produce bias in published results?[J]. Journal of business and psychology, 2019, 34(1): 1-17.
[4] YOUNG S, BANG H. The file-drawer problem, revisited[J]. Science, 2004, 306(5699): 1133-1134.
[5] ROSENTHAL R. The file drawer problem and tolerance for null results[J]. Psychological bulletin, 1979, 86(3): 638-641.
[6] ALLEN D, HOWELL J. Groupthink in science: greed, pathological altruism, ideology, competition, and culture[M]. Cham: Springer, 2020: 99-113.
[7] SMALDINO P, TURNER M, CONTRERAS KALLENS P. Open science and modified funding lotteries can impede the natural selection of bad science[J]. Royal Society open science, 2019, 6(7): 190194.
[8] MCELREATH R, SMALDINO P. Replication, communication, and the population dynamics of scientific discovery[J]. Plos one, 2015, 10(8): e0136088.
[9] WU Y, YANG Y, BRIAN U. A discipline-wide investigation of the replicability of psychology papers over the past two decades[J]. Proceedings of the National Academy of Sciences of the United States of America, 2023, 120(6): e2208863120-e2208863120.
[10] 陈序文, 姚长青, 雷雪. 学术出版视角下科研成果可重复性保障机制研究[J]. 中国科技期刊研究, 2023, 34(11): 1451-1457. (CHEN X W, YAO C Q, LEI X. Reproducibility safeguard mechanism of scientific research results from the perspective of academic publishing[J]. Chinese journal of scientific and technical periodicals, 2023, 34(11): 1451-1457.)
[11] DANIELE F. Is science really facing a reproducibility crisis, and do we need it to?[J]. Proceedings of the National Academy of Sciences of the United States of America, 2018, 115(11): 2628-2631.
[12] 周红霞. 科学研究的可重复性及其保障措施[J]. 科学学研究, 2022, 40(6): 961-968,1104. (ZHOU H X. Practices for reproducible science[J]. Studies in science of science, 2022, 40(6): 961-968,1104.)
[13] Registered Reports: what we’ve learned so far[J]. Nature methods, 2023, 20(10): 1439-1439.
[14] 黄国彬, 刘磊, 陈丽. 注册式研究报告的同行评议机制研究[J]. 图书情报工作, 2021, 65(18): 24-35. (HUANG G B, LIU L, CHEN L. Research on the peer review mechanism of registered reports[J]. Library and information service, 2021, 65(18): 24-35.)
[15] CHAMBERS C, TZAVELLA L. The past, present and future of registered reports[J]. Nature human behaviour, 2022, 6(1): 29-42.
[16] WOZNYJ H, GRENIER K, ROSS R, et al. Results-blind review: a masked crusader for science[J]. European journal of work and organizational psychology, 2018, 27(5): 561-576.
[17] MAGGIN D, ROBERTSON R, COOK B. Introduction to the special series on results-blind peer review: an experimental analysis on editorial recommendations and manuscript evaluations[J]. Behavioral disorders, 2020, 45(4): 195-206.
[18] BUTTON K, BAL L, CLARK A, et al. Preventing the ends from justifying the means: withholding results to address publication bias in peer-review[J]. BMC psychology, 2016, 4(1): 59.
[19] ELMAN C, MAHONEY J, GERRING J. The production of knowledge: enhancing progress in social science[M]. Cambridge: Cambridge University Press, 2020: 221-264.
[20] Registered reports: peer review before results are known to align scientific values and practices [EB/OL]. [2024-10-14]. https://www.cos.io/initiatives/registered-reports.
[21] STERLING T. Publication decisions and their possible effects on inferences drawn from tests of significance—or vice versa[J]. Journal of the American Statistical Association, 1959, 54: 30-34.
[22] LYU X, XU Y, ZHAO X, et al. Beyond psychology: prevalence of p value and confidence interval misinterpretation across different fields[J]. Journal of pacific rim psychology, 2020, 14: e6.
[23] NOSEK B, EBERSOLE C, DEHAVEN A, et al. The preregistration revolution[J]. Proceedings of the National Academy of Sciences of the United States of America, 2018, 115(11): 2600-2606.
[24] FANELLIA D, COSTASB R, IOANNIDISA J. Meta-assessment of bias in science[J]. Proceedings of the National Academy of Sciences of the United States of America, 2017, 114(14): 3714-3719.
[25] 王阳, 肖昆. 论控制偏见的编辑制度革命——关于预注册遏制可重复性危机的机理研究[J]. 科学学研究, 2022, 40(4): 594-601, 664. (WANG Y, XIAO K. On the new editing system revolution to control bias——the studies of mechanism research of pre-registration to keep down reproducibility crisis[J]. Studies in science of science, 2022, 40(4): 594-601, 664.)
[26] POLYZOS N, VALACHIS A, PATAVOUKAS E, et al. Publication bias in reproductive medicine: from the European Society of Human Reproduction and Embryology annual meeting to publication[J]. Human reproduction, 2011, 26(6): 1371-1376.
[27] SCHEEL A, SCHIJEN M, LAKENS D. An excess of positive results: comparing the standard psychology literature with registered reports[J]. Advances in methods and practices in psychological science, 2021, 4(2): 251524592110074.
[28] PEPLOW M. Social sciences suffer from severe publication bias[J]. Nature, 2014, 7(4): 251-257.
[29] The missing pieces: a collection of negative, null and inconclusive results [EB/OL]. [2024-10-14]. https://collections.plos.org/collection/missing-pieces/.
[30] ALLEN C, MEHLER D. Open science challenges, benefits and tips in early career and beyond[J]. Plos biology, 2019, 17(5): e3000246.
[31] 胡传鹏. 将预注册的重复实验纳入心理学研究方法的课程[J]. 心理技术与应用, 2019, 7(5): 261-262, 265. (HU C P. Incorporate pre-registered repeat experiments into the curriculum of psychological research methods[J]. Psychology: techniques and applications, 2019, 7(5): 261-262, 265.)
[32] GREENWALD A. Consequences of prejudice against the null hypothesis[J]. Psychological bulletin, 1975, 82: 1-20.
[33] FANELLI D. “Positive” results increase down the hierarchy of the sciences[J]. Plos one, 2010, 5(5): e10068.
[34] FIELD S, WAGENMAKERS E, KIERS H, et al. The effect of preregistration on trust in empirical research findings: results of a registered report[J]. Royal Society open science, 2020, 7(4): 181351.
[35] COSTA E, INBAR Y, TANNENBAUM D. Do registered reports make scientific findings more believable to the public? [J]. Collabra: psychology, 2022, 8(1): 32607.
[36] 郁林羲, 姚思卉, 康银花. 从科技期刊出版源头推进开放科学运动——TOP Factor的产生及影响[J]. 中国科技期刊研究, 2021, 32(1): 3-13. (YU L X, YAO S H, KANG Y H. Promote the open science movement from the source of the publication of scientific journals: the emergence and influence of top factor[J]. Chinese journal of scientific and technical, 2021, 32(1): 3-13.)
[37] Transparency and openness promotion (top) guidelines[EB/OL]. [2024-10-14]. https://www.cos.io/initiatives/top-guidelines.
[38] World conferences on research integrity: Hong Kong principles [EB/OL]. [2024-10-14]. https://www.wcrif.org/guidance/hong-kong-principles.
[39] BAKER M. 1,500 scientists lift the lid on reproducibility[J]. Nature, 2016, 533(7604): 452-454.
[40] MUNAFÒ M, NOSEK B, BISHOP D, et al. A manifesto for reproducible science[J]. Nature human behaviour, 2017, 1(1): 1-9.
[41] GOODMAN S, FANELLI D, IOANNIDISA J. What does research reproducibility mean? [J]. Science translational medicine, 2016, 8(341): 12-17.
[42] 黄国彬, 郑霞. 推进开放科学发展的注册式研究报告及其价值研究[J]. 图书情报工作, 2020, 64(7): 73-83. (HUANG G B, ZHENG X. Research on the registered reports and its value of promoting open science development[J]. Library and information service, 2020, 64(7): 73-83.)
[43] 王阳, 肖昆. 可重复性危机与预注册新进路[J]. 科学学研究, 2020, 38(5): 779-786. (WANG Y, XIAO K. Reproducibility crisis and new approach of pre-registration[J]. Studies in science of science, 2020, 38(5): 779-786.)
[44] 徐慧慧, 晏华. 基于相对危险度的儿童先心病风险因素分析算法[J]. 计算机科学, 2021, 48(6): 210-214. (XU H H, YAN H. Relative risk degree based risk factor analysis algorithm for congenital heart disease in children[J]. Computer science, 2021, 48(6): 210-214.)
[45] TIKHONOVA E, KIRILLOVA O. Citation culture: citing authors behaviour vs. trust in research results[J]. Science editor and publisher, 2023, 7(2): 166-181.
[46] DUYX B, URLINGS M, SWAEN G, et al. Citation bias: questionable research practice or scientific misconduct?[J]. Journal of the Royal Society of Medicine, 2022, 115(1): 31-35.
[47] FANELLI D. Positive results receive more citations, but only in some disciplines[J]. Scientometrics, 2013, 94(2): 701-709.
[48] 杨卫, 刘细文, 黄金霞, 等. 构筑开放科学行动路线图把握开放科学发展机遇[J]. 中国科学院院刊, 2023, 38(6): 783-794. (YANG W, LIU X W, HUANG J X, et al. Building roadmap for open science action:seizing global development opportunities[J]. Bulletin of Chinese Academy of Sciences, 2023, 38(6): 783-794.)
[49] 贺颖, 徐小然. 基于特征数据的开放式同行评议关键因素的实证分析[J]. 中国科技期刊研究, 2022, 33(8): 1041-1045. (HE Y, XU X R. Empirical research on key factors of open peer review: based on characteristic data[J]. Chinese journal of scientific and technical, 2022, 33(8): 1041-1045.)
[50] 心理学开放科学苏州倡议[J]. 心理与行为研究, 2023, 21(5): 720. (Open Science Suzhou Initiative in Psychology[J]. Studies of psychology and behavior, 2023, 21(5): 720.)
[51] Reproducibility and replicability in science national academies[EB/OL]. [2024-10-14]. https://www.nationalacademies.org/our-work/reproducibility-and-replicability-in-science.
Options
文章导航

/

〈 〉