INF5220 8

1
INF5220 - 8

• Lecture 12th of January 2006

2
Silverman on quality

• Aim of this course is to instill in you a
  methodological awareness
• Question Should we just trust qualitative
  researchers? (eg. based on their political
  credentials and demonstration of involvement)
• Or is there a basic, shared attempt (within the
  research field) to generate credible and valid
  knowledge?
• Credibility the extent to which any research
  claim has been shown to be based on evidence
• We will discuss
• Validity
• Reliability
• Generalisation

3
Validity and reliability

• Validity has to do with what we might call
  truthfulness
• Reliability has to do with consistency of
  measurements
• Example repeated readings of two termometers in
  boiling water (100 ºC)
• A is reliable, but gives invalid results
• B is unreliable, but gives relatively valid
  results
• Are your procedures realiable, are your
  conclusions valid?

4
Validity (1)

• Is the account true?
• Are the deviant or contrary cases excluded?
• Is a glossy picture presented?
• The common problem of anecdotalism
• that a few exemplary instances are offered,
  without reasons for selecting them, or a
  discussion of the typicality or
  representativeness of them.
• Claims to validity should be based on attempts at
  refutation
• Avoid jumping to conclusions just because there
  is some evidence that seem to lead in an
  interesting direction. Make an effort to falsify
  your initial assumptions about the data

5
Validity (2)

• Often advocated strategies (but flawed in S
  view)
• Triangulation (e.g. combination of different
  methods that give you different but complementary
  data)
• Remember that results from methods are situated,
  context-bound
• Respondent validation (take your accounts and
  interpretations back to the subjects)
• But do they have a privileged epistomological
  status?
• Silvermans advice
• The refutability principle
• Constant comparative method
• Comprehensive data treatment
• Deviant case analysis
• Using appropriate tabulation

6
Reliability (1)

• Reliability refers to the degree of consistency
  with which instances are assigned to the same
  category by different observers or by the same
  observer on different occasions (Hammersley,
  1992 87)
• Consistency can be along several dimensions
  between different observers, between the results
  of different data collection methods, across
  different cases, over time, etc.

7
Reliability (2)

• Important to document the procedure. How do you
  present your data?
• Use low-inference descriptors, e.g. verbatim
  accounts of what people said, extracts from field
  notes.
• On the contrary, high-inference descriptors may
  be your polished account of what you see in
  your material, your interpretations.
• Give details on the relevant context of
  observations and how you recorded and handled the
  notes.
• Have you standardised your data
  collection/construction to some degree? Describe
  your field-note conventions (if you used
  standardised templates for data collection, your
  research groups practices for sharing and
  discussion of analytic memos, etc.)

8
Generalisability (1)

• Quantitative research sampling is about
  selecting a representative subsection of a
  population. A good choice helps establish the
  representativeness and thus the generalisability.
  
• This is not so straightforward in Qualitative
  research, but we may think about
  representativeness in several ways
• Complement qualitative studies with quantitative
  measures
• Purposive sampling think critically about the
  parameters of the population before you select
  case
• Theoretical sampling your attempt is not to
  generalise to populations, but you select samples
  based on their potential to offer interesting
  theoretical insights
• Usually a single case, chosed for convenience
  reasons (e.g. access)

9
Generalisability (2)

• The validity of inferences drawn from on e of
  more cases does not depend on the
  representativeness of the cases in a statistical
  sense, but on the plausibility and cogency of
  the logical reasoning used in describing the
  results from the cases, and in drawing
  conclusions from them (Klein and Myers, citing
  Walsham)
• Four types of generalisations (Walsham, 1995)
• The development of concepts
• The generation of theory
• The drawing of specific implications
• The contribution of rich insights

10
What is high quality in research?

• Silvermans criteria (see tables 15.1, 15.2,
  15.3). To what degree does it match the list of
  criteria that I gave you for the Master thesis
  evaluation?
• Silverman suggest four basic criteria
• build useful theories,
• be self-critical,
• selecting appropriate research methods,
• make a practical contribution.

11
Klein and Myers

• Criteria for conducting and evaluating
  interpretive research
• Based on the perspective of hermeneutics
• Seven principles (page 72)
• The fundamental principle of the hermeneutic
  circle
• The principle of contextualisation
• The principle of interaction between the
  researcher and the subjects
• The principle of abstraction and generalization
• The principle of dialogical reasoning
• The principle of multiple interpretations
• The principle of suspicion

• http//www.misq.org/archivist/bestpaper/misq99.pdf

12
Some other methodological concepts you may
encounter

• Quantitative and/or positivist research
• Dependent (outcome variable, what you measure)
  and independent variables (factors, what you
  manipulate or change)
• External validity related to generalizations (to
  which degree would your conclusions hold for
  other people in other places, at other times?)
• Internal validity (for studies of causal
  relationships). How well have you managed to
  separate out and distinguish the effects among
  the variables? Are there other possible causes
  (explanations) for your observations that you
  have forgotten or neglected?

13
The handouts from last week

• Examples and templates that you may use (modify
  if needed)
• Template for field notes
• A description of how to create analytic memos
• Template for research proposal
• Master thesis structure
• Fieldwork report (5-10 pages)
• Content
• Facts where did you go, what time, whom did
  you see?
• What happened? Describe what you saw in your
  observations, what was being discussed in the
  interviews etc. (the findings)
• Sum up What did you learn (that you did not know
  before)?
• Reflect on what implications may this have for
  further work? (does it help you towards
  formulating a more focused research question? Do
  you know more about what you do NOT want to do?)
• Not analysis, and/or use of theory
• Due by March 1st (or negotiable).

14
Other issues

• How to get help while writing your thesis (e.g.
  to locate relevant literature)
• Use readings from previous courses, and
  additional literature
• Ask faculty to suggest relevant literature (both
  supervisor and others)
• How to plan in order to succeed
• Serious thinking and literature survey now
• Serious work with proposal during the spring
• Start serious writing by Dec 1st)

15
Topic for discussion

• What does it mean to be scientific?
• One answer adopting methods of study that are
  appropriate to the subject matter
• From Petters email
• Theory as maps?
• Is sociological/anthropological knowledge useful
  in IS?
• About hypotheses