Research Quality Framework

1
Research Quality Framework

• Research Quality Framework Briefing
• University of Ballarat Staff

2
Agenda

• Introduction Prof. Wayne Robinson
• Overview and explanation of RQF Prof. Warren
  Payne
• Research Data Pack Ms. Jan Watson
• Web-based data collection tool Mr. Sasha Ivkovic
• Questions

3
Research Quality Framework

• Overview and Explanation of the RQF
• Professor Warren Payne

4
RQF Presentation Outline

• UB Research Performance2000-2004
• Context What is the RQF?
• Developments to date
• RQF Trial Timetable
• Overview of RQF processes
• RQF Trial process
• Examples

5
RQF Presentation Outline

• UB Research Performance2000-2004
• Context What is the RQF?
• Developments to date
• RQF Trial Timetable
• Overview of RQF processes
• RQF Trial process
• Examples

6
UB Research Performance2000-2004

• Research Income
• 2000 1.9M
• 2001 2.7M
• 2002 3.3M
• 2003 3.8M
• 2004 4.4M

• Publications
• 2000 65
• 2001 85
• 2002 117
• 2003 163
• 2004 165

7
RQF Presentation Outline

• UB Research Performance2000-2004
• Context What is the RQF?
• Developments to date
• RQF Trial Timetable
• Overview of RQF processes
• RQF Trial process
• Examples

8
Context What is the RQF?

• The RQF framework is designed to ensure the
  Commonwealth research funding is being used to
  support areas of research excellence and greatest
  public benefit (Nelson, 2005).

9
Context What is the RQF?

• External Context
• Change in Federal government policy regarding the
  funding of university research.
• RQF data will provide the basis for the
  distribution of research funding, to ensure that
  areas of the highest quality research are
  rewarded (Nelson, 2005).
• Will involve all of the Institutional Grants
  Scheme (IGS) and at least 50 of the Research
  Training Scheme (RTS).

10
Context What is the RQF

• Internal Context
• RQF will hopefully enable us to
• obtain valuable information and data on research
  quality across key research groups, and
• assist strategic decision making for the
  development of research within UB.

11
RQF Presentation Outline

• UB Research Performance2000-2004
• Context What is the RQF?
• Developments to date
• RQF Trial Timetable
• Overview of RQF processes
• RQF Trial process
• Examples

12
Developments to date

• DEST published The Preferred Model in September
  2005
• Responses to model made by universities and AVCC
• RQF cycles proposed
• 1. 2007 (1/1/2001 31/12/2006)
• 2. 2010 (1/1/2004 31/12/2009)
• 3. 2016 (1/1/2010 31/12/2015)

13
Developments to Date

• University groupings conducting trials of the
  RQF process in 2006 in preparation for the
  introduction of the full process in 2007.
• UB is part of the New Generation Universities
  grouping.

14
Developments to Date

• New Generation Universities (NGUs)
• Southern Cross University
• University of Western Sydney
• University of Ballarat
• Victoria University
• Central Queensland University
• University of the Sunshine Coast
• Edith Cowan University
• University of Canberra
• Australian Catholic University

15
Developments to Date

• Trial process
• UB RQF processes
• Measurement,
• Moderation,
• Feedback.
• NGU processes
• Moderation.

16
RQF Presentation Outline

• UB Research Performance2000-2004
• Context What is the RQF?
• Developments to date
• RQF Trial Timetable
• Overview of RQF processes
• RQF Trial process
• Examples

17
UNIVERSITY OF BALLARAT RESEARCH QUALITY
ASSESSMENT Timetable Draft
18
UNIVERSITY OF BALLARAT RESEARCH QUALITY
ASSESSMENT Timetable Draft
19
RQF Presentation Outline

• UB Research Performance2000-2004
• Context What is the RQF?
• Developments to date
• RQF Trial Timetable
• Overview of RQF processes
• RQF Trial process
• Examples

20
Overview of RQF processes

• Measurement
• What is measured?
• Quality, impact, esteem
• Which measures?
• Quantitative and qualitative measures
• Which researchers?
• Institutions to choose
• Proportion to be reported
• What level of aggregation?
• Research groups/clusters

21
Overview of RQF processes

• Measurement
• Evidence Portfolios
• 4 highest quality outputs- books, journals,
  patents etc
• 4 impacts of the research - economic, social,
  environmental, cultural, academic
• 4 esteem measures
• Document research effort of the Centre/Cluster

22
Overview of RQF processes

• Measurement
• Research Outputs (Scored 1-5)
• 5 Pioneering outstanding performance
• 4 International Standard very strong
  performance
• 3 Excellent National Standard strong
  performance
• 2 Average National Standard moderate
  performance
• 1 Below National Standard below average
  performance

23
Overview of RQF processes

• Measurement
• Research Outputs (Scored 1-5)
• 5 Pioneering outstanding performance
• Groundbreaking, highly innovative research that
  is world-renown. Notable intellectual or
  creative advance. No more than 5 of research
  outputs across a discipline or field of research
  or across a large institution can be expected to
  warrant this assessment.

24
Overview of RQF processes

• Measurement
• Research Outputs (Scored 1-5)
• 4 International Standard very strong
  performance
• Research considered truly internationally
  competitive and making a major contribution to
  the advancement of knowledge. No more than 15
  of research outputs across a discipline or field
  of research or across a large institution can be
  expected to warrant this assessment or the higher
  (5) assessment.

25
Overview of RQF processes

• Measurement
• Research Outputs (Scored 1-5)
• 3 Exceeding National Standard strong
  performance
• Research of high standard and making a solid
  incremental contribution to knowledge in the
  field. Around 30 of all research outputs across
  a discipline or field of research or across a
  large institution can be expected to warrant this
  assessment or the higher (5 or 4) assessment.

26
Overview of RQF processes

• Measurement
• Research Outputs (Scored 1-5)
• 2 Average National Standard moderate
  performance
• Research of average/moderate standard but not at
  the level of significantly advancing knowledge in
  the field. Around half of all research outputs
  across a discipline or field of research or
  across a large institution can be expected to
  warrant this assessment.

27
Overview of RQF processes

• Measurement
• Research Outputs (Scored 1-5)
• 1 Below National Standard below average
  performance
• Research of low standard no contribution to
  understanding or insight demonstrated.

28
Overview of RQF processes

• Measurement
• Research Impact (Scored High, Moderate Limited)
• High Fundamentally altered policy or practice in
  particular field, or produced a major,
  identifiable social, economic, industrial or
  environmental change locally or internationally.
• Moderate Significantly altered policy or
  practice in particular field, or produced a
  major, identifiable social, economic, industrial
  or environmental change.
• Limited Little or no identifiable change in
  policy or practice.

29
Overview of RQF processes

• Measurement
• Research Peer Esteem (Scored 1-5)
• 5 Pioneering outstanding performance
• 4 International Standard very strong
  performance
• 3 Excellent National Standard strong
  performance
• 2 Average National Standard moderate
  performance
• 1 Below National Standard below average
  performance

30
Overview of RQF processes
31
Overview of RQF processes

• Moderation
• Proposed Groupings
• Biological and cellular science and biotechnology
• Public health and health services
• Physical, chemical and earth sciences
• Psychology, neurological, behavioural and
  cognitive sciences
• Engineering
• Social sciences, law, education, politics,
  sociology and indigenous studies
• Mathematical and information sciences and
  technology
• Economics, commerce, management and information
  management
• Agricultural, veterinary, food and environmental
  sciences, architecture, urban environment and
  building
• Humanities
• Clinical sciences and clinical physiology
• Creative and performing arts and design

32
Overview of RQF processes

• UB Moderation
• Establish UB Moderation Panels
• Maximum of 12, likely to be some integration.
• Comprise of
• UB RQF coordinator(s)
• UB Cluster nominees (eg. Centre Director, HOS)
• External experts

33
Overview of RQF processes

• NGU Moderation
• Work with 9 other NGUs
• Assess across NGUs
• Develop benchmarks
• Evaluate benchmarks
• Who assesses?
• National experts
• International experts
• Stakeholders
• What discipline clusters?
• 12 based on RFCD codes

34
Overview of RQF processes

• Feedback
• Outcomes provided to individuals (confidential)
  and clusters

35
Overview of RQF processes

• Ministers Expert Advisory Group Next Steps
• Four panels developing further details
• Funding models
• Research Impact
• Assessment Panels
• Eligibility and Quality Criteria
• Report in late December to Minister

36
RQF Presentation Outline

• UB Research Performance2000-2004
• Context What is the RQF?
• Developments to date
• RQF Trial Timetable
• Overview of RQF processes
• RQF Trial process
• Examples

37
RQF Trial process

• 3 Stage Process
• Self Assessment process
• Centre/Cluster Review
• Mock RQF with full portfolios

38
RQF Trial process

• Self Assessment
• All active researchers participate
• About 1 day to complete
• Electronic submission process
• When March 2006

39
RQF Trial process

• Centre/Cluster Review
• Develop assessment process and guidelines
• Self Assessment tool used
• Evidence portfolio produced
• Aggregate to the Centre/Cluster level
• Use the learning to refine future processes

40
RQF Trial process

• Mock RQF with full portfolios
• Across 9 NGU Universities
• Portfolios prepared
• NGU assessment panels formed
• Feedback provided to researchers
• When March to end Nov 2006
• Ready for RQF in 2007

41
RQF Presentation Outline

• UB Research Performance2000-2004
• Context What is the RQF?
• Developments to date
• RQF Trial Timetable
• Overview of RQF processes
• RQF Trial process
• Examples

42
Examples

• Output
• Esteem
• Impact

43
Examples
44
Examples

• Output
• Description/Type Refereed Journal
• Baca, M., Miller, M. and Slamin, Vertex-magic
  total labelings of generalized Petersen graphs.
  (2002). Int. J. of Computer Mathematics.
  791259-1264.
• Journal has an Impact rating of xxx. The
  publication has been cited yyy times.

45
Examples

• Output
• Description/Type Chapter in book
• Wellard, S.J., Beddoes, L. (2005).
  Constructions of chronic illness. In C.
  Rogers-Clark, K Martin-McDonald A McCathy
  (Eds.) Living with Illness Psychosocial
  Challenges. A text for nurses and other caring
  professionals. McLennan Petty.
• Describes the new advances and approaches to
  nursing. Contribution to publication xx.

46
Examples

• Output
• Description/Type Exhibitions and performances
• A solo exhibition of original Art Fields of
  Time at the invitation of the National Gallery.
  The work was exhibited for 3 months and was the
  exhibited by invitation at galleries in Mildura
  and Horsham.

47
Examples

• Esteem
• Description/Type International collaborations
• Collaborator in the International Commission of
  Rural Community Sustainability, member of
  steering committee for measurement tool
  development and participant in the Asia/Pacific
  projects.

48
Examples

49
Examples

• Impact
• Description/Type Employment
• Research into the factors affecting the retention
  rate of workers in rural nursing. Resulted in a
  positive increase in retention over 5 years that
  has resulted in a saving to the sector in
  training costs of 2 million per annum and an
  increase in rural nurses by 30.

50
Examples

• Impact
• Description/Type Community participation
  Effects of Self Esteem on Health of the Aging
  population
• From 2000-2005 on behalf of the Victorian
  Department of Human Services I participated in a
  team of researchers and health care workers in
  implementing the research findings from research
  on Impacts of self esteem in reducing illness
  and poor wellbeing in the Aging population at a
  state level. We trained 200 healthcare workers in
  the use of techniques to build self esteem.

51
Examples

• Impact
• Description/Type Capacity Building Small
  Business Performance
• Over the period I have lead a small team in
  informing and working with small businesses in
  Victoria to improve management skills. This
  involved 50 people in workshops and 300 people
  attending training courses. The impact of this
  work has been a reduction in staff turnover of
  participating businesses of 50. Based on
  estimated turnover rates in the sector this
  program if adopted at 25 of eligible businesses
  would result in a saving of 12 m per annum in
  recruitment costs to the sector.

52
Conclusions

• Formal categorisation of universities
• Shift of resources across sector
• Shift from global to discipline based comparisons
  
• Increased public awareness of research
  intensiveness
• Flow on to student recruitment
• Staff recruitment and retention

53
References

• Thorn, C. (2005). Research Quality Framework
  presentation. Edith Cowan University
• DEST (2005). Research Quality Framework
  Assessing the quality and impact of research in
  Australia The preferred model.
  http//www.dest.gov.au/sectors/research_sector/pol
  icies_issues_reviews/key_issues/research_quality_f
  ramework/rqf_preferred_model.htmThe_RQF_Preferred
  _Model
• AVCC (2005) Research Quality Framework The AVCC
  reponse to the EAC preferred model.
  http//www.avcc.edu.au/documents/publications/poli
  cy/submissions/RQF-The-AVCC-Response-2005.pdf

54
Any Questions

• ?