收稿日期: 2016-09-18
修回日期: 2016-12-13
网络出版日期: 2017-01-05
基金资助
本文系国家自然科学基金项目“力学学科发展态势评估研究：2006-2015”（项目编号：L1524029）研究成果之一。
Comparison of the Role of Peer Review and Bibliometrics in Research Evaluation
Received date: 2016-09-18
Revised date: 2016-12-13
Online published: 2017-01-05
[目的/意义] 科研评价实践中普遍使用的两种评价手段和量度工具是同行评议与计量为基础的评价（文献计量、经济计量甚至替代计量）。为了对比同行评议质量判断与计量为基础的评价结果的一致性，进而回答同行评议能否被新兴的计量评价方法取代。[方法/过程] 本文采用非实证的研究方式，收集前人在论文发表、项目资助、研发评价实践、个人颁奖以及Altmetrics计量评价领域应用两种评价方法对研发绩效测度的120多篇实证研究进行元分析并汇总成定量化综述，基于认知演化的视角进行系统记叙。[结果/结论] 传统同行评议为基础的决策判断始终是研发评价的首要量度工具且地位无法取代，它与引用为基础的计量评价结果在统计上仅存在适度的正相关性，Spearman秩相关系数r∈[0.24，0.88]，地位上后者更应充当前者的补充而非替代，即建构在定量辅助基础上的知情同行评议模式不仅使评价结果更加客观透明，也添加额外的附加价值，通过数据来加深对问题的认识，为评价的黄金法则。
万昊 , 谭宗颖 , 朱相丽 . 同行评议与文献计量在科研评价中的作用分析比较[J]. 图书情报工作, 2017 , 61(1) : 134 -152 . DOI: 10.13266/j.issn.0252-3116.2017.01.016
[Purpose/significance] The two most commonly used measuring instruments for scientific assessment are peer review and metric-based scientific evaluation(Bibliometrics, Econometrics and Altmetrics).[Method/process] In order to explore the consistency between the above two measuring "tools" and answer the question about whether the peer review can be replaced by the emerging metric-based evaluation, this paper adopts a non-empirical research approach and collects more than 120 studies on the R&D performance measurements through the two methods at the level of institution, project, paper, and awarding for the quantitative review with meta-analysis, following the evolutionary perspective for systematic narration.[Result/conclusion] Peer review is still the most important tool for scientific assessment always and forever, but the potential of citation-based bibliometrics is huge. Quantitive evaluation should serve as the supplementary rather than substitutionof qualitative peer review judgement with moderateSpearman rank correlationcoefficient r∈[0.24, 0.88]. The complementary combination of the qualitative and quantitative evaluations will make the evaluation more objective and transparent with additional value through data manifestation to better understand the problem,which is called the "Golden Standard"of evaluation.
Key words： R&D evaluation; peer review; bibliometri; cmetric; correlation
[1] DE SOLLA PRICE D J.Little science, big science-and beyond[M].New York:Columbia University Press, 1986.
[2] KUHN T S. The structure of scientific revolutions[M]. Chicago:University of Chicago Press, 1962.
[3] FEIST G J. The psychology of science and the origins of the scientific mind[M]. New Haven:Yale University Press, 2006.
[4] HICKS D. Performance-based university research funding systems[J]. Research policy, 2012, 41(2):251-261.
[5] BERTOUT C, SCHNEIDER P. Editorship and peer-review at A&A[J]. Astronomy & astrophysics, 2004, 420(3):E1-E14.
[6] Publishing Research Consortium. Peer review in scholarly journals:perspective of the scholarly community-an international study[R]. Bristol:The Consortium, 2008.
[7] WESSELY S, WOOD F. Peer review of grant applications:asystematic review[M]//GodleeF, Jefferson T. Peer review in health sciences. London:British Medical Journal Books, 1999.
[8] 杨振宁, 王贻芳. 中国今天是否应该建造超大对撞机?[EB/OL].[2016-09-13]. http://www.shobserver.com/news/detail?id=29763.
[9] HOFFMANN H, JOYE D, KUHN F, et al. Der SNF im spiegel der forschenden:synthesebericht[M]. Neuchãtel:Schweizerischer Informations und Datenarchivdienst für die Sozialwissenschaften (SIDOS), 2002.
[10] HORNBOSTEL S, OLBRECHT M. Peer review in der DFG:die fachkollegiaten[M]. Bonn:Institut für Forschungsinformation und Qualitätssicherung, 2007.
[11] LI D, AGHA L. Big names or big ideas:do peer review panels select the best science proposals?[J]. Science, 2015, 348(6233):434-438.
[12] BORNMANN L. Scientific peer review[J]. Annual review of information science and technology, 2011, 45:199-245.
[13] LAMONT M. How professors think:inside the curious world of academic judgment[M]. Cambridge:Harvard University Press, 2009.
[14] MCCLELLAN J E. Specialist control:the publications committee of of the Académie Royale Des Sciences (Paris) 1700-1793[M]. Philadelphia:American Philosophical Society, 2003.
[15] SILER K, LEE K, BERO L. Measuring the effectiveness of scientific gatekeeping[J]. Proceedings of the National Academy of Sciences, 2015, 112(2):360-365.
[16] GEISLER E. The metrics of science and technology[M]. Westport:Quorum Books, 2000.
[17] CAMPANARIO J M. Peer review for journals as it stands today:part 1[J]. Science communication, 1998, 19(3):181-211.
[18] MARSH H W, JAYASINGHE U W, BOND N W. Improving the peer-review process for grant applications:reliability, validity, bias, and generalizability[J]. American psychologist, 2008, 63(3):160-168.
[19] Wellcome Trust. Review of Wellcome Trust Ph.D. research training:career paths of a 1988-1990 prize student cohort[M]. London:The Trust, 2001.
[20] MOED H F, HALEVI G. Multidimensional assessment of scholarly research impact[J]. Journal of the Association for Information Science and Technology, 2015, 66(10):1988-2002.
[21] ABRAMO G, D'ANGELO C A. Evaluating research:from informed peer review to bibliometrics[J]. Scientometrics, 2011, 87(3):499-514.
[22] GARFIELD E. Citation indexes for science-A new dimension in documentation through association of ideas[J]. Science, 1955, 122(3159):108-111.
[23] GARFIELD E. Premature discovery or delayed recognition-Why?[J]. Current contents, 1980, (21):5-10.
[24] VAN RAAN A F J. Sleeping beauties in science[J]. Scientometrics, 2004, 59(3):467-472.
[25] VAN RAAN A F J. Avanced bibliometric methods as quantitive core of peer review based evaluation and foresight exercises[J]. Scientometrics, 1996, 36(3):397-420.
[26] MEHO L I, YANG K. Impact of data sources on citation counts and rankings of LIS faculty:Web of science versus scopus and google scholar[J]. Journal of the American Society for Information Science and Technology, 2007, 58(13):2105-2125.
[27] HIRSCH J E. An index to quantify an individual's scientific research output[J]. Proceedings of the National Academy of Sciences of the United States of America. 2005, 102(46):16569-16572.
[28] MOED H F. Citation analysis in research evaluation[M]. Dordrecht:Springer, 2005.
[29] NARIN F. Evaluative bibliometrics:the use of publication and citation analysis in the evaluation of scientific activity[M]. Washington DC:National Science Foundation, 1976.
[30] GARFIELD E. Would mendels work have been ignored if Science Citation Index was Available 100 Years Ago[J]. Current contents, 1969, 12(47):69-70.
[31] RINIA E J, VAN LEEUWEN TH N, VAN VUREN H G, et al. Comparative analysis of a set of bibliometric indicators and central peer review criteria:Evaluation of condensed matter physics in the Netherlands[J]. Research policy, 1998, 27(1):95-107.
[32] RINIA E J, VAN LEEUWEN TH N, VAN VUREN H G, et al. Influence of interdisciplinarity on peer-review and bibliometric evaluations in physics research[J]. Research policy, 2001, 30(3):357-361.
[33] VAN RAAN A F J. Comparison of the Hirsch-index with standard bibliometric indicators and with peer judgment for 147 chemistry research groups[J]. Scientometrics, 2006, 67(3):491-502.
[34] WALTMAN L, VAN ECK N J, VAN LEEUWEN TH N, et al. On the correlation between bibliometric indicators and peer review:reply to Opthof and Leydesdorff[J]. Scientometrics, 2011, 88(3):1017-1022.
[35] SEGLEN P O. Citation rates and journal impact factors are not suitable for evaluation of research[J]. Acta orthopaedica scandinavica, 1998, 69(3):224-229.
[36] MACROBERTS M H, MACROBERTS B R. Problems of citation analysis:a critical review[J]. Journal of the American Society for InformationScience, 1989, 40(5):342-342.
[37] MACROBERTS M H, MACROBERTS B R. Problems of citation analysis[J]. Scientometrics, 1996, 36(3):435-444.
[38] MACROBERTS M H, MACROBERTS B R. Problems of citation analysis:a study of uncited and seldom-cited influences[J]. Journal of the American Society for Information Science and Technology, 2010, 61(1):1-12.
[39] BROWMAN H I, STERGIOU K I. The use and misuse of bibliometric indices in evaluating scholarly performance[J]. Ethics in science and environmental politics, 2008, 8(1):1-107.
[40] WARNER J. A critical review of the application of citation studies to the research assessment exercises[J]. Journal of Information Science, 2000, 26(6):453-459.
[41] SUGIMOTO C R, LARIVIÈRE V, NI C. Journal acceptance rates:across-disciplinary analysis of variability and relationships with journal measures[J]. Journal of informetrics, 2013, 7(4):897-906.
[42] GLÄNZEL W. Seven myths in bibliometrics about facts and fiction quantitative science studies[J]. ISSI Newsletter, 2008, 2(1):24-32.
[43] COLE S, COLE J R. Scientific output and recognition:astudy in operation of reward system in science[J]. American sociological review, 1967, 32(3):377-390.
[44] KATZ J S. The self-similar science system[J]. Research policy, 1999, 28(5):501-517.
[45] NEDERHOF A J, VAN RAAN A F J. Citation theory and the Ortega hypothesis[J]. Scientometrics, 1987, 12(5):325-328.
[46] CAMACHO-MINANO M D M, NUNEZ-NICKEL M. The multilayered nature of reference selection[J]. Journal of the American Society for Information Science and Technology, 2009, 60(4):754-777.
[47] VAN NOORDEN R, MAHER B, NUZZO R. The top 100 papers[J]. Nature, 2014, 514(7524):550-553.
[48] IOANNIDIS J P A, BOYACK K W, SMALL H, et al. Bibliometrics:is your most cited work your best?[J]. Nature, 2014, 514(7524):561-562.
[49] DANIEL HD. Guardians of science:fairness and reliability of peer review[M]. Weinheim:Wiley-VCH, 2004.
[50] BRAUN T, GLÄNZEL W, SCHUBERT A. Scientometric indicators:a32 country comparison of publication productivity and citation impact[M]. Philadelphia:World Scientific, 1985.
[51] GREENBERG S A. How citation distortions create unfounded authority:analysis of a citation network[J]. British Medical Journal, 2009, 339(7716):210-213.
[52] WEINGART P. Evaluation of research performance:the danger of numbers[M].Jülich:Forschungszentrum Jülich, 2003.
[53] CATALINI C, LACETERA N, OETTL A. The incidenceand role of negative citations in science[J]. Proceedings of the National Academy of Sciences of the United States of America, 2015, 112(45):13823-13826.
[54] 靖继鹏, 马费成, 张向先. 情报科学理论[M]. 北京:科学出版社, 2009.
[55] NAGEL E. The structure of science:problems in the logic of scientific explanation[M]. London:Routledge & Kegan Paul, 1961.
[56] VAN RAAN A F J. On growth, ageing, and fractal differentiation of science[J]. Scientometrics, 2000, 47(2):347-362.
[57] 王文娟, 马建霞, 陈春, 等.引文文本分类与实现方法研究综述[J]. 图书情报工作, 2016, 60(6):118-127.
[58] DE SOLLA PRICE D J. A general theory of bibliometric and other cumulative advantage processes[J]. Journal of the American Society for Information Science, 1976, 27(5-6):292-306.
[59] MORAVCSIK M J, MURUGESAN P. Some results on function and quality of citations[J]. Social studies of science, 1975, 5(1):86-92.
[60] MARTENS B V D, GOODRUM A A. The diffusion of theories:a functional approach[J]. Journal of the American Society for Information Science and Technology, 2006, 57(3):330-341.
[61] 刘盛博, 王博, 唐德龙, 等.基于引用内容的论文影响力研究——以诺贝尔奖获得者论文为例[J]. 图书情报工作, 2015, 59(24):109-114.
[62] PENDLEBURY D A. The use and misuse of journal metrics and other citation indicators[J]. Scientometrics, 2009, 57(1):1-11.
[63] BRAUN T, GLÄNZEL W, SCHUBERT A. A Hirsch-type index for journals[J]. Scientometrics, 2006, 69(1):169-173.
[64] DE SOLLA PRICE D J. Networks of scientific papers[J]. Science, 1965, 149(3683):510-515.
[65] NEWMAN M E J. The structure and function of complex networks[J]. Society for industrial and applied mathematics review, 2003, 45(2):167-256.
[66] 万昊, 谭宗颖, 鲁晶晶, 等. 2001-2014 年引文分析领域发展演化综述[J]. 图书情报工作, 2015, 59(6):120-136.
[67] HAITUN S D. Stationary scientometric distributions. 2. Non-gaussian nature of scientific activities[J]. Scientometrics, 1982, 4(2):89-104.
[68] BURRELL Q L. The use of the generalized waring process in modelling informetric data[J]. Scientometrics, 2005, 64(3):247-270.
[69] EGGHE L. A new short proof of Naranan's Theorem, explaining Lotka's Law and Zipf's Law[J]. Journal of the American Society for Information Science and Technology, 2010, 61(12):2581-2583.
[70] 马费成, 刘向. 知识网络的演化(I):增长与老化动态[J]. 情报学报, 2011, 30(8):787-795.
[71] 万昊, 谭宗颖, 朱相丽, 等.科学知识增长过程中系统自组织创生模式研究[J]. 图书情报工作, 2015, 59(24):93-101.
[72] BÖRNER K, CHEN C M, BOYACK K W. Visualizing knowledge domains[J]. Annual review of information science and technology, 2003, 37:179-255.
[73] CHEN C M, CHEN Y, HOROWITZA M et al. Towards an explanatory and computational theory of scientific discovery[J]. Journal of informetrics, 2009, 3(3):191-209.
[74] 王晓光. 科学知识网络的形成与演化(Ⅱ):共词网络可视化与增长动力学[J]. 情报学报, 2010, 29(2):314-322.
[75] LOCK S. A difficult balance:editorial peer review in medicine[M]. Philadelphia:ISI Press, 1985.
[76] OPTHOF T, FURSTNER F, VAN GEER M, et al. Regrets or no regrets? no regrets! the fate of rejected manuscripts[J]. Cardiovascular research, 1985, 45(1):255-258.
[77] MCDONALD R J, CLOFT H J, KALLMES D F. Fate of manuscripts previously rejected by the American Journal of Neuroradiology:a follow-up analysis[J]. American journal of neuroradiology, 2009, 30(2):253-256.
[78] BORNMANN L, MARX W, SCHIER H, et al. From black box to white box at open access journals:predictive validity of manuscript reviewing and editorial decisions at Atmospheric Chemistry and Physics[J]. Research evaluation, 2010, 19(2):105-118.
[79] BORNMANN L, DANIEL HD. Selecting manuscripts for a high impact journal through peer review:a citation analysis of communications that were accepted by Angewandte Chemie International Edition, or rejected but published elsewhere[J]. Journal of the American Society for Information Science and Technology, 2008, 59(11):1841-1852.
[80] BORNMANN L, MUTZ R, MARX W, et al. A multilevel modelling approach to investigating the predictive validity of editorial decisions:do the editors of a high profile journal select manuscripts that are highly cited after publication?[J]. Journal of the Royal Statistical Society Series A-Statistics in Society, 2011, 174(4):857-879.
[81] RAGONE A, MIRYLENKA K, CASATI F, et al. A quantitative analysis of peer review[C]//Proceedings of ISSI 2011:the 13th conference of the international society for scientometrics and Informetrics. Belgium:International Society for Scientometrics and Informetrics, 2011:724-736.
[82] 金碧辉, 汪寿阳, 任胜利, 等. 论期刊影响因子与论文学术质量的关系[J]. 中国科技期刊研究, 2000, 11(4):202-205.
[83] 王一华. 基于IF(JCR)、IF(Scopus)、H指数、SJR值、SNIP值的期刊评价研究[J]. 图书情报工作, 2011, 55(16):144-148.
[84] MAVIS B, KATZ M. Evaluation of a program supporting scholarly productivity for new investigators[J]. Academic medicine, 2003, 78(7):757-765.
[85] BORNMANN L, DANIEL HD. Committee peer review at an international research foundation:predictive validity and fairness of selection decisions on post-graduate fellowship applications[J]. Research Evaluation, 2005, 14(1):15-20.
[86] BORNMANN L, DANIEL HD. Criteria used by a peer review committee for selection of research fellows. abooleanprobit analysis[J]. International journal of selection and assessment, 2005, 13(4):296-303.
[87] BORNMANN L, DANIEL HD. Selection of research fellowship recipients by committee peer review. analysisof reliability, fairness and predictive validity of Board of Trustees' decisions[J]. Scientometrics, 2005, 63(2):297-320.
[88] BORNMANN L, DANIEL HD. Selecting scientific excellence through committee peer review-A citation analysis of publications previously published to approval or rejection of post-doctoral research fellowship applicants[J]. Scientometrics, 2006, 68(3):427-440.
[89] BORNMANN L, DANIEL HD. Convergent validation of peer review decisions using the H index. Extent of and reasons for type I and type Ⅱ errors[J]. Journal of informetrics, 2007, 1(3):204-213.
[90] BORNMANN L, WALLON G, LEDIN A. Does the committee peer review select the best applicants for funding? An investigation of the selection process for two European Molecular Biology Organization programmes[J]. PLoS ONE, 2008, 3(10):e3480.
[91] REINHART M. Peer review of grant applications in biology and medicine. Reliability, fairness, and validity[J].Scientometrics, 2009, 81(3):789-809.
[92] NEUFELD J, VON INS M. Informed peer review and uninformed bibliometrics?[J]. Research evaluation, 2011, 20(1):31-46.
[93] ARMSTRONG P W, CAVERSON M M, ADAMS L, et al. Evaluation of the Heart and Stroke Foundation of Canada Research Scholarship Program:research productivity and impact[J]. Canadian journal of cardiology, 1997, 13(5):507-516.
[94] CHAPMAN G B, MCCAULEY C. Predictive validity of quality ratings of National Science Foundation graduate fellows[J]. Educational and psychological measurement, 1994, 54(2):428-438.
[95] MELIN G, DANELL R. The top eight percent:development of approved and rejected applicants for a prestigious grant in Sweden[J]. Science and public policy, 2006, 33(10):702-712.
[96] HORNBOSTEL S, BÖHMER S, KLINGSPORN B, et al. Funding of young scientist and scientific excellence[J]. Scientometrics, 2009, 79(1):171-190.
[97] VAN DEN BESSELAAR P, LEYDESDORFF L. Past performance as predictor of successful grant applications. a case study[M]. Den Haag:Rathenau Instituut, 2007.
[98] VAN DEN BESSELAAR P, LEYDESDORFF L. Past performance, peer review, and project selection:acase study in the social and behavioral sciences[J]. Research evaluation, 2009, 18(4):273-288.
[99] BORNMANN L, LEYDESDORFF L, VAN DEN BESSELAAR P. A meta-evaluation of scientific research proposals:different ways of comparing rejected to awarded applications[J]. Journal of informetrics, 2010, 4(3):211-220.
[100] 段志光, 卢祖洵, 王彤. 诺贝尔生理学或医学奖获得者学术影响力研究[J]. 科学学研究, 2007, 25(2):222-227.
[101] 鲍玉芳, 马建霞. 诺贝尔奖与科学家论文数量、被引频次的相关性——基于2000-2010年诺贝尔化学、物理学获奖者的实证研究[J]. 图书馆理论与实践, 2015(8):40-45.
[102] EGGHE L, GUNS R, ROUSSEAU R. Thoughts on uncitedness:Nobel laureates and fields medalists as case studies[J]. Journal of the American Society for Information Science and Technology, 2011, 62(8):1637-1644.
[103] 科研圈. 汤森路透发布2016诺奖预测,两位华人科学家入选[EB/OL].[2016-09-21]. http://mt.sohu.com/20160921/n468892890.shtml.
[104] AKSNES D W, TAXT R E. Peer reviews and bibliometric indicators:acomparative study at a Norwegian university[J]. Research evaluation, 2004, 13(1):33-41.
[105] OPTHOF T, LEYDESDORFF L. A comment to the paper by Waltman et al. Scientometrics, 87, 467-481[J]. Scientometrics, 2011, 88(3):1011-1016.
[106] OPPENHEIM C, NORRIS M. Citation counts and the research assessment exercise V:archaeology and the 2001 RAE[J]. Journal of documentation, 2003, 59(6):709-730.
[107] BUTLER L, MCALLISTER I. Metrics or peer review? evaluating the 2001 UK research assessment exercise in political science[J]. Political studies review, 2009, 7(1):3-17.
[108] REALE E, BARBARA A, COSTANTINI A. Peer review for the evaluation of academic research:lessons from the Italian experience[J]. Research evaluation, 2007, 16(3):216-228.
[109] ABRAMO G, D'ANGELO C A, CAPRASECCA A. Allocative efficiency in public research funding:can bibliometrics help?[J]. Research policy, 2009, 38(1):206-215.
[110] FRANCESCHET M, COSTANTINI A. The first Italian research assessment exercise:abibliometricperspective[J]. Journal of informetrics, 2011, 5(2):275-291.
[111] LI XM, THELWALL M. F1000, Mendeley and traditional bibliometric indicators[C]//International conference on science and technology indicators. 2012:451-551.
[112] 杨柳,陈贡. Altmetrics 视角下科研机构影响力评价指标的相关性研究[J]. 图书情报工作, 2015, 59(15):106-114, 132.
[113] BORNMANN L, LEYDESDORFF L. The validation of (advanced) bibliometric indicators through peer assessments:a comparative study using data from InCites and F1000[J]. Journal of informetrics, 2013, 7(2):286-291.
[114] THELWALL M, FAIRCLOUGH R. The influence of time and discipline on the magnitude of correlations between citation counts and quality scores[J]. Journal of informetrics, 2015, 9(3):529-541.
[115] 宋丽萍, 王建芳. 基于F1000与WoS的同行评议与文献计量相关性研究[J]. 中国图书馆学报, 2012, 38(3):63-69.
[116] 由庆斌, 韦博, 汤珊红. 基于补充计量学的论文影响力评价模型构建[J]. 图书情报工作, 2014, 58(11):5-11.
[117] BORNMANN L. Inter-rater reliability and convergent validity of F1000Prime peer review[J]. Journal of the Association for Information Science and Technology, 2015, 66(12):2415-2426.
[118] WALTMAN L, COSTAS R. F1000 recommendations as a potential new data source for research evaluation:acomparison with citations[J]. Journal of the Association for Information Science and Technology, 2014, 65(3):433-445.
[119] WILEY H S. Peer review isn't perfect. But it's not a conspiracy designed to maintain the status quo[J]. The scientist, 2008, 22(11):31.
[120] DU J, TANG X L, WU Y S. The effects of research level and article type on the differences between citation metrics and F1000 recommendations[J]. Journal of the Association for Information Science and Technology, 2016, 67(12):3008-3021.
[121] WESSELY S. Peer review of grant applications:What do we know[J]. Lancet, 1998, 352(9124):301-305.
[122] VAN RAAN A F J. Fatal attraction:conceptual and methodological problems in the ranking of universities by bibliometric methods[J]. Scientometrics, 2005, 62(1):133-143.
[123] Higher Education Funding Council for England. Report on the pilot exercise to develop bibliometric indicators for the research excellence framework[EB/OL].[2016-01-02]. http://www.hefce.ac.uk/pubs/hefce/2009/09_39/09_39.pdf.
[124] MOED H F. The future of research evaluation rests with an intelligent combination of advanced metrics and transparent peer review[J]. Science and public policy, 2007, 34(8):575-583.
[125] BRAUN T. How to improve the use of metrics[J]. Nature, 2010, 465(6):870-872.
[126] 鲁索, 全薇.期刊影响因子,旧金山宣言和莱顿宣言:评论和意见[J]. 图书情报知识, 2016 (1):4-14.
/
| 〈 |
|
〉 |
