Measuring Impact: Issues and Opportunities

1
Measuring Impact Issues and Opportunities

• Dr John R Gibbins
• University of Newcastle
• and Chair of YNE Hub

2
Aims

• To identify the drivers
• To explore some methodological issues
• To check what is available nationally - PRES
• To suggest some opportunities for simple
  measurement, evaluation and impact systems for
  local consumption
• To help inform a debate at the Roberts Forum at
  Manchester
• To emphasise the potential of Alumni relations
  for getting the best feedback on impact

3
Drivers for KP1s

• Our need to know we are achieving our goals
  (individually and collectively)
• Our need to know how to improve our performance
  (individually and collectively)
• To justify our expenditure to the Treasury other
  stakeholders including employers
• To make a good case for continuation and
  development of the Roberts Agenda

4
Difficulties and Problems

• Which of the above is the priority outcome?
• Academics prefer the first two - softer
• RCUK and Treasury the latter - harder
• If the second two take priority is it possible to
  comply?
• What are the indicators and measures?
• Can they be measured?
• Can we locate an appropriate data base?
• Can we isolate the variables in terms of impact?

5
Harder or Softer?(Chris Parks slide)

• Managerial
• Evidence-based decision-making
• Performance management KPIs
• Focus external, compliance and accountability
• Softer approach
• Evidence-based decision-making
• Understanding of processes and impacts
• Focus internal, student experience and
  enhancement

6
KPIsAny performance indicators must fulfil the
following needs

• They must measure what needs to be indicated
• They must be measurable
• They must have credibility with all stakeholders
• They should be the smallest number possible
• They must be clear
• They must allow both efficient and effective
  implementation and operation of the scheme

7
With these in mind the indicators proposed should
be

• Small in number
• Simple in operation
• Helpful to managers in enhancing their practices
• Many HEIs have small Roberts income and have
  to carry the same burden of review. We need to
  keep it simple!

8
Rugby Team Approachfor the Nation

• Softer than harder approach
• A compendium approach
• Secondary data analysis i.e. it makes use of
  existing data
• Some New Data Collection revised PRES survey
  Reading Survey (Pamela Denicola)

9
Suggested Approach for the Regions

• Harder (quantitative) rather than softer
  (qualitative) easier to collect metrics data
• Focused rather then the compendium something a
  lot easier
• Primary data collection rather than secondary
  that gives HEIs a simple task list for data
  collection built on existing HESA requirements

10
What needs measuring?

• Indicators of Impact
• Access
• Engagement
• Impact
• Is quality an extra indicator? No. Its what the
  other 3 indicate as well as impact

11
Access

• Access are all researchers, despite mode of
  study, able to access a programme of
  developmental activities in the funded HEI or
  unit?
• Access is coming into the presence of

12
Engagement

• Engagement do all researchers take up, own and
  engage the programme of activities in the funded
  HEI? Is personal and professional development
  owned by the researcher? Do researchers evidence
  their development?
• Engagement is involved and interested, partisan

13
Impact

• Impact is their evidence that the accessed and
  engaged programmes have enhanced the skills of
  the researcher to meet the JSS skills
  requirement, and the explicit requirements of
  employers and other stakeholders needs?
• An impression to be made upon.

14
Possible Measures of AccessBold are recommended

• Range of access systems i.e. numbers and types
  (more the better?)
• Number and range of services available to
  students (taken from a drop down list)
• Numbers () of students accessing these services
  e.g. eBooking and Blackboard data
• Numbers () owning a Programme Handbook in
  whatever form
• Attendance data at Inductions ()
• Numbers () accessing Grad web based services.

15
Possible measures of engagement?

• Numbers () attending activities
• Numbers () of part-time students attending
  development sessions
• Numbers () completing an ePortfolio or
  equivalent
• Feedback and evaluation data () plus 4.00 on the
  postgraduate equivalent of National Student
  Survey data
• Participation () engaging Grad national,
  regional, local activities

16
Possible measures of Impact

• Survey data on the attitudes and satisfaction of
  Employers
• Survey data on the attitudes and satisfaction of
  Alumni
• Supervisors satisfaction data
• Graduate School date on submissions and
  completions
• Careers data on job destinations

17
An Impact Survey?Can we measure employer
satisfaction?

• Careers services have been trying this for
  decades with varying success
• Focus Groups of Alumni fade away, fatigue
• Focus groups of Employers (WDPDo Empress and
  Sheffield) issues of low response, overload,
  ignorance of in the field information
• Destination Surveys issues of data bases,
  response, reliability
• Consultation with Professional Bodies,
  Federations, Regional Employment Sector Groups

18
PRES and READING

• HEA have developed the BOS/CROS Survey for
  Postdocs to Postgraduates as well
• Voluntary national survey not for benchmarking
• http//www.heacademy.ac.uk/3919.htm
• www.npc.org.uk/page/1155575518.pdf

19
What is PRES?

• Student focused
• Sector Questionnaire
• Conducted online
• Core plus local questions 4 Scales
• Intellectual climate and environment
• Infrastructure and departmental context
• Supervision
• Skills Development

20
Reading Survey

• Qualitative rather than quantitative
• Interviews with stakeholders including
• -Researchers
• -Supervisors
• -Research managers
• -Deans

21
Gap Analysis

• Inappropriate indicators?
• Misses out on those stakeholders that could
  report on impact
• Focuses on those stakeholders making inputs not
  on those who experience the outputs
• Looks at process not product
• Using inappropriate measures?

22
Alumni

• They experience employment
• Know if they are developed for task
• Know if they have a career
• Know if they have and the names of employers
• Know who in an organisation may be able to
  evaluate skills levels and gaps
• They are known (or should be) to us

23
An Option to ExploreAn Alumni and their Employer
Survey

• An Agenda item at Newcastle University builds on
  Alumni allegiance to the University (under
  development)
• Careers and Alumni to improve their data base of
  PG Researchers (do it faster and fuller) this
  will take some initial investment
• Research unit (CURDs) to draft a questionnaire
  not difficult PRES type questions can be used
• Data Collection to all completing researchers in
  2007 and possibly earlier cohorts

24
Methodology

• Ownership model to improve response rate
• Build on Alumni allegiance to the University to
  get responses
• Build on Alumni line managers duty to their
  colleague to get responses
• So after completing a questionnaire an employer
  section is passed to the employer. Alumni
  customer service.

25
Requirements

• Baseline data before 2007
• Longitudinal collection two yearly
• Conformity in institutions
• HESA requirement from HEIs
• Central encouragement

26
Challenges

• Completing cohorts may be too small to get
  significance solutions are to aggregate years
  aggregate HEIs in regions.
• Response rates may be low solutions in the
  methodology.
• The identified impact cant be attributed to the
  Roberts Skills Activities

27
Response

• If not avoided we still have data on satisfaction
  of users of Roberts skills development cohorts
  which is useful at the local level for
  monitoring, evaluating and improving skills
  programmes
• We can do gap analysis on what employers feel is
  missing and so gap fill
• We have made an effort to meet stakeholder
  accountancy requirements

28
Activities at home

• Ask Careers and Alumni to improve databases of
  Alumni ( rather than employers)
• Consider creating a regional consortium to create
  a viable data base of completing researchers
• Discuss this with Grad, RCUK, HEA, HERD, and
  other involved bodies

29
Conclusions on Impact Survey

• Alumni Surveys offer hard evidence
• Simply and easily Careers duty already
• At relatively low cost to HEIs could be shared
  in consortium
• With reasonable expectations of validity and
  success
• Working with Alumni has many other benefits as
  well

30
Conclusions on KPIs

• Targeted KPIs could be more useful and valid
  than the compendium
• Reducing the KPIs and their measures to a small
  number will reduce local burdens, save time,
  money and duplication
• By making it national via HESA,it will create
  equity and fairness in the community