HomeTown Competitiveness: Conducting Phase I Assessment

1
HomeTown CompetitivenessConducting Phase I
Assessment

• Jennifer Bott, PhD, SPHR
• Associate Professor of Management
• Miller College of Business
• Ball State University

2
Agenda

• Background on HTC Assessment
• Goals of Assessment
• Types of Assessments
• Focus Groups and interviews
• Designing surveys
• Archival data analysis
• Basic analyses and decision making
• Resources

3
Introduction to Assessment
4
Background on Assessment

• Phase I Assessment phase typically lasts up to
  four months
• Community conducts local assessments of
  development history, context, issues and assets
• Create customized strategic opportunities
• Leads into implementation of the strategic
  development plan in Phase II

5
Assessment Goals

• Determine perspectives of community stakeholders
  on chosen pillar within the community
• Entrepreneurship
• Charitable Assets
• Youth Engagement
• Leadership

6
Assessment Outcome

• End product of the assessment
• Collection of data that direct community team
  toward path of action
• Summary of current thinking in community on
  current status of selected pillar
• Launching assessment with community should engage
  them more in the process, outcome and
  implementation of the strategic plan

7
Types of Assessments

• Traditional measures of assessment
• Focus groups
• Interviews
• Surveys (telephonic, online, mail, intercept)
• Other opportunities
• Re-analysis of previously-collected data

8
Types of Data

• Two primary types of data emerge from
  assessments
• Qualitative
• Verbal responses to open or closed-ended
  questions
• Quantitative
• Numeric responses to (usually) closed-ended
  questions

9
Choice of Assessment(s)

• Assessment method should
• Provide most insight into questions at hand
• Involve stakeholders
• Adhere to following criteria
• Reliable
• Valid
• Cost-effective
• Feasible
• Combinations of assessment methods may be most
  beneficial

10
Focus Groups and Interviews
11
Focus Groups

• Characteristics
• 7 10 members
• Share some relationship to topic of interest
• Represent divergent opinions on topic and
  typically represent different stakeholders
• Repeated several times with different groups of
  people
• Requires collection of qualitative data

12
Topics for Focus Groups

• Strategic planning
• Needs assessment
• Current state analysis
• Program design
• Perspectives on a singular issue

13
Uses of Focus Groups

• Focus groups are especially useful
• Exploratory, preliminary study
• Gap in understanding or communication
• Uncover data related to complex behavior
• Discover interactive synergies between
  participants
• Uncover additional information leading to a
  large-scale study effort
• High value placed on open-ended data collection

14
When NOT to Use Focus Groups

• Focus groups are not recommended for
• Emotionally-charged subject
• Researcher/HTC team not in control of the
  environment or message
• Team expects statistical summaries from data
• Is not economical to achieve data collection
  needed
• Confidentiality cant be guaranteed

15
Outcomes of Focus Groups

• Categorized data from focus groups can be used
• As stand-alone, qualitative information
• As starting point for development of a
  quantitative measure

16
Designing Successful Focus Groups

• Create safe environment
• May require use of a neutral facilitator
• Create a plan for focus group that details
• Purpose
• Who to include
• Participants
• Number of sessions
• Incentives for participation (if necessary)
• Location

17
Focus Group Questions

• Categories of questions
• Opening
• Everyone answers quickly (10 - 20 seconds)
• Introductory
• General topic of discussion, foster conversation
• Transition
• Approach the focal questions
• Key
• Drive the study 2 to 5 questions that require
  most attention
• Ending
• Bring closure and allow for reflection

18
Focus Group Questions (contd)

• In most cases, focus group questions are
  open-ended
• What are our greatest weaknesses in youth
  engagement?
• Closed-ended questions are not appropriate for
  sparking discussion
• Would you participate in a training success?
• Avoid asking why in response to a participants
  answer

19
Analyzing Focus Group Data

• Determining analysis strategy
• Least time intensive Memory based
• Report is created immediately based on
  facilitators memory
• Most time intensive Transcript based
• Combines field notes with transcript to write
  final report
• Most common Note based
• Relies on field notes to create focus group
  report

20
Focus Group Report

• The focus group report summarizes
• Participants
• Setting
• Key questions
• Themes that emerged from key question responses
• Limitations
• Recommendations

21
Conducting Interviews

• Interviews can also be used as assessment tools
• One-on-one or panel (2 or more)
• Collect qualitative data
• Can be used as singular collection method or in
  tandem with focus groups or surveys

22
Uses of Interviews

• Appropriate when
• Group perspectives are not valuable
• Topic is too sensitive to discuss in group
  setting
• Want to collect limited information (e.g., only
  want to speak to a few individuals)
• Focus groups are not feasible

23
Surveys
24
Uses of Surveys

• Surveys are appropriate when
• Know the parameters of the content to be assessed
• Have identified a sample that will provide
  meaningful data
• Need quantitative figures (rather than
  qualitative descriptions)
• Can be coupled easily with qualitative data
  collection

25
Considerations with Survey Administration and
Design

• Decision points on survey design
• Sample
• Who will respond?
• How many?
• Administration mode
• Telephone
• Intercept (in-person)
• Snail mail
• Internet
• Necessary response rates

26
Survey Questionnaire Development

• Six basic steps
• Decide what information to collect
• Decide what type of questionnaire should be used
• Develop first draft
• Revise questionnaire
• Pilot test
• Edit questionnaire and specify methods of
  collection

27
Question Writing Guidelines

• Characteristics of good items
• Simple, direct, familiar language
• Clear and specific questions
• Do not use leading, loaded, or double-barreled
  questions
• Use short statements (20 words or less)
• Edit for readability

28
Designing the Scale

• Most common response options
• Agreement
• Frequency
• Evaluation
• Number of response categories
• 3 to 5
• Neutral point good or bad idea?

29
Evaluate Wording

• Which is a better question?
• Age?
• What was your age on your last birthday?
• Is there a problem with this item?
• What do you like best about this neighborhood?
  (Were interested in anything, like houses, the
  people, the parks, or whatever.)
• Do you favor or oppose gun control legislation?

30
Item Stem Creation

• Agreement items Declarative
• I am uncomfortable around strangers.
• Frequency items Events, behaviors
• You exercise strenuously enough to raise your
  heart rate.
• Evaluation items Persons, places, things
• How well your favorite sports team played last
  week.

31
Designing the Survey

• Some simple rules for the ordering and design of
  questions
• Keep items together if they use the same
  responses
• Dont overfill each page with questions
• Avoid including too many different response
  options
• Pilot test with individuals similar to those
  being sampled

32
Importance of Instructions

• Give direction for response for those unfamiliar
  with format
• Create a common frame of reference
• The following questions concern peoples
  opinions and beliefs about jobs and careers.
  These questions refer to jobs in general and not
  the job you presently have or a particular job
  you once had. (Spector, 1992)

33
Archival Data
34
Using Archival Data

• In some cases, data has already been collected
  regarding your pillar of interest
• Ask groups that are associated with your pillar
  (e.g., Chamber of Commerce, local schools,
  philanthropic groups)
• Can be either quantitative (surveys) or
  qualitative (interviews, news coverage, etc.)

35
Possible Analyses with Archival Data

• Content analysis
• Assessment of trends
• Analysis of communications

36
Limitations

• Archival data has many advantages and
  disadvantages
• Advantages
• Data is already collected (cheap, easy to use)
• Data may already be analyzed
• Disadvantages
• Cant be tailored to fit needs
• Inability to determine the reliability of the
  methods used to collect data

37
Analysis and Decision Making
38
Conclusion of the Assessment

• Use the assessment data to identify key
  findings about the pillar assessment
• A key finding is a full sentence used to
  summarize learning.
• Including a statistics strengthens this finding.
• 73 of high school students surveyed stated that
  they would return to our community if
  opportunities exist.

39
Conclusion (contd)

• Key findings are used to drive the development of
  the problems to be solved at the end of Phase I.

40
Tangible Outcomes of Assessment

• A written report should be provided to
  individuals who participated in the survey
• Details overall findings (at the group level
  only)
• May provide information about uses of the
  assessment
• Depends on confidentiality of the process
• Can hopefully meet a need of the sample

41
Moving to Decision Making

• Totality of evidence from both qualitative and
  quantitative assessment efforts should be
  examined
• Looking for evidence that supports teams beliefs
  or uncovers new, unexpected information
• Evidence should create a logical next step in
  assessment or implementation

42
Resources
43
Resources to Aid in the Assessment Process

• Center for Business and Economic Research at Ball
  State University
• Partnered with Building Better Communities
• Professional consultants
• Other HTC Communities

44
Resources (contd)

• Books
• Survey Research Methods (4th ed.), Floyd Fowler
  (Sage Publications, 39.95)
• Focus Groups A Practical Guide for Applied
  Research, Krueger and Casey (Sage Publications,
  45.32)

45
Links to Nebraska Surveys

• http//www.zoomerang.com/Shared/SharedResultsSurve
  yResultsPage.aspx?IDL22Z2S57GK54
• http//www.zoomerang.com/Shared/SharedResultsSurve
  yResultsPage.aspx?IDL23KD72VDQVC
• http//www.zoomerang.com/Shared/SharedResultsSurve
  yResultsPage.aspx?IDL23KDC4HWP6J