Multi-Dimensional Factors in Academic Research Evaluation

1
Multi-Dimensional Factors in Academic Research
Evaluation

• Xiaodong Zhang
• Ohio State University

2
Impact of Strong Research to Universities

• A foundation of improving undergraduate and
  graduate programs
• Timely updating contents in classrooms
• Creating research environment and opportunities
  to students
• Bringing competitiveness and recognition
• New knowledge, discoveries, and innovations.
• Continue to attract strong students, excellent
  faculty, and research grants.

3
How is Research Measured?

• Research Production is Typically Measured by
• Publications in refereed conferences and journals
• Software and prototypes
• Invited lectures
• Patents
• Research Awards and honors
• Input of Research
• Research proposal writing, and research project
  planing
• Research grants
• Recruiting and forming teams.

4
How is Research Impact Measured?

• Research results are useful
• Citations of the publications
• Widely used software and prototypes
• Technology transfer from original research
  results
• Useful patents
• Influence in the research community
• Leading new directions in the field
• The quality of research is essential
• Number of publications should not be a major
  metric
• Our research should be Impact-driven

5
Where to Publish?

• Why publish in top venues?
• Each field has its own flagship and leading
  conferences/journals defined by reputations
• Most researchers only read papers there.
• Conferences versus Journals
• In certain fields, such as systems, architecture,
  networking, and databases, top conference papers
  are prestigious, highly visible, with 4-5 or more
  reviews in depth, and low acceptance rate.
• SCI and EI should NOT be used as a guidance
• An SCI/EI entry does not necessarily reflect its
  quality in the field
• The quality should be judged by the peers in the
  field.
• The impact factor sometimes is field-size
  dependent.

6
Establishing Identity and Reputation

• Conducting research on focused topics in depth
• Making names behind some important innovations
• US NSF Career Program for Junior Faculty
• A 5 year research/education plan, aiming at
  conducting research to solve a small set
  significant problems.
• A strong department should have several
  identities
• Paying attention to building on existing
  strength.
• Rewarding to strong research identity/reputation

7
What is an Acceptable Research Production?

• How to quantify research productions?
• Number vs. quality of publications
• Research grants/expenditure (mission vs. basic
  research)
• Supervising Ph.D. students productivity and
  quality of placements.
• Impact of research after many years.
• Conference/journal organizations
• Not part of the research, but professional
  service.
• Each department has its formula
• Resources dependent (teaching load, et. al.)

8
Effects of University Rankings

• Rankings by society have to be a consideration
• Rankings significantly affect student recruiting
• University rankings influence parents
• Faculty recruiting is also affected.
• However, these rankings are a reference not an
  absolute indicator.

9
Major Sources of University Ranking in US

• US News World Reports (University ranking)
• Published every August.
• Based on 6 criteria to quantify comprehensive
  ability.
• Quality of undergraduate students is an important
  consideration.
• US News World Reports (Graduate school ranking)
• Published every March.
• Different school rankings are based on different
  disciplines.
• Quality of graduate programs is the major factor.
• National Science Foundation (funding statistics)
• Research expenditure of last year published every
  August
• Ranking the research scale of universities.
• National Research Council (Department ranking)
• Published every 10 to 15 years
• Ranking all the Ph.D. granted departments in all
  majors

10
US News World Reports 6 Criteria

• Undergraduate student selectivity
• SAT scores, high school ranking, and other
  activities.
• Acceptance rate.
• Faculty academic reputation
• Research activities and accomplishments
• Faculty-student ratio
• Reflecting the efforts and attention to students,
  e.g. class size.
• Student retention and graduation rate
• Seriousness of the students and management of
  curriculum.
• Financial resources
• University revenue and endowed funds.
• Alumni donation rate
• Reflecting graduates proud, passion, and care to
  the university.

11
Major Themes of Graduate Program Ranking

• Scholarships and impacts
• Quality of faculty publications and their
  influence to the field
• Citations rates and technology transfers
• Ph.D. student production and placement
• How many per year, and where they go after
  graduations.
• Ph.D. alumni achievements
• The group distinguishing themselves in the
  fields.
• Faculty resources
• Award winners, members of the academies, and
  others.
• Research scale
• Measured by the amount of expenditure by external
  grants

12
Foundations of Academic Excellence

• Excellent undergraduate programs
• Highly selected students with comprehensive
  talents.
• Providing highly quality education and research
  environment.
• All top U.S. universities have top undergraduate
  programs.
• Excellent research and education infrastructure
• Students are able to effectively learn and
  research.
• Excellent faculty
• World-class professors who are well established
  in their fields.
• High standards in research and education
• Rigorous training to students and doing impact
  driven research.
• Ambitious students who will play leadership
  anywhere
• Enter grow the wisdom Depart serve the better
  country and the kind.
• Should not be too narrowly focused.

13
Obstacles to Establish World-Class Universities
in China

• Student selection system has a lot of limits
• The exam system narrowly focuses on knowledge
  learning
• Selected best students may not be innovative
  and ambitious.
• Research is too much economy development driven
• Too many mission oriented projects with low
  research values
• Faculty selection system is not rigorous and
  rewarding
• Not very selective, and not as prestigious as
  administrators.
• SCI papers driven work will never lead to world
  class.
• Only impact driven work is the foundation of
  scholarships.
• The centralized control limits our vision and
  steps
• Lacking freedom and competitions, and peer
  reviews.
• A selection of university president other leading
  administrators is tightly controlled
• When can we openly select university leaders
  worldwide?

14
Mr. Hus 4 Conditions of an Established
University

• In 1947, Hu Shi urged the government to invest 5
  existing universities to make them be
  established and independent in the world
  within 10 years.
• 1. Providing first-class education environment.
• 2. Being able to retain young researchers.
• 3. Being able to solve countrys problems in
  science, industries, health, and national
  defense.
• 4. Being able to collaborate with foreign
  scientists to solve open and hard problems.

15
Considerations of Building World Class
Universities

• High selectivity of students far beyond exams
• Building high quality faculty life term
  scholars.
• Research standard should be defined by peers not
  by upper administrations for easy management.
• Basic research projects should not be driven by
  economic development missions.
• Basic research grants should be dominant in
  universities, not mission oriented funds.
• University governance should be professors based.
• Recruiting presidents, provosts, deans, and
  department chairs should be open to the world.