Peer Review Can Make Metrics Accountable ?????????????

1
Peer Review Can Make Metrics Accountable?????????
????

• Kelli Barr
• ????
• Department of Philosophy and Religion Studies
• Center for the Study of Interdisciplinarity
• University of North Texas
• ???????
• ???????
• ???????
• Translation by ZHANG Wei

2
The Question at Hand?????

• Are metrics better tools for evaluating research
  than peer review?
• ???????,???????????????

• Are metrics better tools for evaluating research
  than peer review?
• Increasingly, academics and administrators are
  answering with an unqualified yes
• ???????????????????????,????????
• H-index, g-index, webometrics
• H???g????????
• US News and World Report, Shanghai Rankings
• ?????????,??(????)????
• National research evaluations
• ??????
• Hiring, promotion, and tenure decisions
• ??????????????

Peer Review Can Make Metrics Accountable Interna
tional Workshop on Peer Review Dalian University
of Technology Monday, May 21 Wednesday, May 23,
2012
3
Peer Review Bibliometrics????

????? Subjective
??? Contingent
??? Non-transparent
???? Unaccountable ????? Costly
????
Objective ??? Replicable ???? Transparent ??? Acc
ountable ???? Low(er) Cost ????
Peer Review Can Make Metrics Accountable Interna
tional Workshop on Peer Review Dalian University
of Technology Monday, May 21 Wednesday, May 23,
2012
4

• Defining the dichotomy using these terms creates
  a distinction without much of a difference
• ????????????????????????????
• Bibliometrics are ultimately based on peer review
  processes
• ?????????????????????
• All metrics instantiate specific values that
  affect the outcome of the measurements
• ??????????????????,???????????

Peer Review Can Make Metrics Accountable Interna
tional Workshop on Peer Review Dalian University
of Technology Monday, May 21 Wednesday, May 23,
2012
5
Forbes concerned with economic, financial,
commercial, and business outcomes ??? ??????????
??????
Academic ranking of world universities
concerned with bibliometric (research)
outcomes ????????? ????????(??)??
http//chronicle.com/article/30-Ways-to-Rate-a-Col
lege/124160/
6

• Journal impact factor, calculated using Web of
  Knowledge
• ????????Web of Knowledge???
• Sciences Nature 34.48
• ???? ??
• Cell 31.152
• ??
• Science 29.747
• ??
• Social Sciences J. of Economic Lit. 6.919
• ???? ??????
• Political Analysis 3. 756
• ????
• Ann. Rev. of Sociology 3.702
• ?????
• Philosophy Am. J. of Bioethics 4.000
• ?? ?????????
• Phil. and Public Affairs 1.957
• ???????

• Journal impact factor, calculated using Web of
  Knowledge
• Sciences Nature 34.48
• Cell 31.152
• Science 29.747
• Social Sciences J. of Economic Lit. 6.919
• Political Analysis 3. 756
• Ann. Rev. of Sociology 3.702
• Philosophy Am. J. of Bioethics 4.000
• Phil. and Public Affairs 1.957

Do social science and philosophy journals really
have that much less impact? ????????????????????
???
Peer Review Can Make Metrics Accountable Interna
tional Workshop on Peer Review Dalian University
of Technology Monday, May 21 Wednesday, May 23,
2012
7

• Bibliometrics assume that all research conforms
  to the model of scientific research
• ????????????????????????
• Cumulative
• ???
• Disciplinary
• ???

• Bibliometrics assume that all research conforms
  to the model of scientific research
• Cumulative
• Disciplinary
• Bibliometrics highlight certain kinds of
  research
• ????????????????
• Journal publications
• ????
• Quantitative
• ???
• Methodologically focused
• ??????
• Review papers
• ????

Peer Review Can Make Metrics Accountable Interna
tional Workshop on Peer Review Dalian University
of Technology Monday, May 21 Wednesday, May 23,
2012
8
Implications ??

• Bias toward disciplinary, status quo research
• ??????,????
• Reinforcing the status quo is not morally neutral
  
• ???????????????
• Appeals to the supposed objectivity,
  transparency, etc. of metrics are used as
  evidence of their normative superiority to peer
  review
• ???????????????????????????????????????????

Peer Review Can Make Metrics Accountable Interna
tional Workshop on Peer Review Dalian University
of Technology Monday, May 21 Wednesday, May 23,
2012
9
Peer Review Can Make Metrics Accountable Interna
tional Workshop on Peer Review Dalian University
of Technology Monday, May 21 Wednesday, May 23,
2012
10

• Accountability is about rational justification
• ??????????
• Definitions vary widely with context
• ??????????????????
• Legal liability
• ????
• Return on investment
• ??????
• Academic rigor
• ??????

• Accountability is about rational justification
• ??????????
• Definitions vary widely with context
• Common denominator evaluating whether or not
  current tactics meet overall strategic goals
• ???????????????????????

• Accountability is about rational justification
• ??????????
• Definitions vary widely with context
• Common denominator evaluating whether or not
  current tactics meet overall strategic goals
• Accountability is more than just rule following
• ???????????

Peer Review Can Make Metrics Accountable Interna
tional Workshop on Peer Review Dalian University
of Technology Monday, May 21 Wednesday, May 23,
2012
11

• Conclusions
• ??
• Neither metrics nor peer review are an
  unqualified good
• ???????????????
• The question of the meaning of accountability is
  also the question of the meaning of
  responsibility
• ?????????????????????
• If science funding is to be a goal-directed
  activity, then evaluations of the science funded
  cannot normatively privilege values such as
  objectivity for their own sake.
• ??????????????????,???????????????????????????????
  ?????????

Peer Review Can Make Metrics Accountable Interna
tional Workshop on Peer Review Dalian University
of Technology Monday, May 21 Wednesday, May 23,
2012