Title: Relevance in academic research
1Relevance in academic research
- Dan Remenyi PhD
- Visiting Professor
- School of Systems and Data Studies
- Trinity College Dublin
- dan.remenyi_at_tcd.ie
2Establishing the context
- The issues of relevance and rigor are complex
concepts and need careful understanding. - Some words have a large semantic footprint and
relevance, rigor, validity, reliability,
generalisability and integrity are good examples
of these. - No one can proceed with any understanding of
research without having a clear agreement with
regards the meaning of terms.
3The effectiveness and efficiency parallel
- A quick rule of thumb understanding of relevance
is doing the right thing or looking at the right
issues - A quick rule of thumb understanding of rigor is
doing the thing right - These are the colloquialisms coined by Peter
Drucker to explain effectiveness and efficiency - Thus relevance equates to effectiveness and rigor
relates to well not so much efficiency but
something more like adequacy
4Relevance and Rigor
- Relevance is a high level issue in that you can
say that the research project is relevant or not
relevant as a whole it could also be said
that the use of case studies was not relevant to
the objectives of the study - Rigor is an issue which permeates the whole
project in that you can say that the literature
review was not rigorously approached or the
sample was not rigorously chosen, or the
conclusions were not rigorously written up.
5Academic Research is a Challenge
6Academic research
- What is the nature of academic research?
- It is an intellectual activity
- It is an attempt to answer a question
- It is a process of question, data and analysis
- Academic research has two primary attributes-
- It adds something of value to the body of
theoretical knowledge.and practice! - It demonstrate a high level of scholarship
7Something of value to the body of theoretical
knowledge.and practice
- Something of value is reflected in the concept of
relevance. - Relevance needs to be perceived by both the
non-academic stakeholders and also the academic
community. - Without some contribution to theory the research
will not be much valued by academics. - Mode 2 brings both together.
8Scholarship and rigor
- Being well read and being able to demonstrate
that in conversation - Being able to craft an argument in writing
- Scholarship is connected to rigor
- An argument is a series of logical steps by which
someone is persuaded. In this sense is not
confrontational. - Academic work i.e. either a dissertation or a
peer reviewed paper needs to argue for relevance,
rigor, validity, reliability, generalisability.
9Quality academic research
- Dimensions of quality in academic research
- addresses a pertinent question
- uses academic language with care - remember the
fog factor - uses appropriate data
- is written up in intelligible English and correct
format - is presented in an attractive way - structure
- invoking authority
- finds an acceptable/useful answer
- Being parsimonious quality academic research
needs to be relevant and rigorous
10Double hurdle
- Academic research needs to be relevant i.e. of
interest and of use to a wide range of management
at different levels and rigorous i.e. display
academic excellence. - Being both relevant and rigorous has been
described as a synthesis strategy.
11Research question
- The research question has been reached through a
careful examination of and understanding of - The field of study
- The topic
- The specific questions and sub-questions
12Some critical issues
- Something of value
- As judged by the academic community
- Body of theoretical knowledge
- Add something to established domains
- Parsimonious
- Expressed in a precise and economical way
- Authority
- Ideas/concepts/theories which have through the
peer review process - Relevance
- Has value to a group of stakeholders
- Rigor
- Complies with the rules of the research paradigm
13Three more issues
- Validity ..Has the research addressed the
intended issues i.e. addressed the issues which
are required to answer the research question? - Reliability..Has the research been conducted
sufficiently well (rigorously) to provide results
in which we can have confidence? - Generalisabilty..To what extent can the
research be applied else where?
14Rationalism versus Empiricism
- Rationalism (theory)
- The mind is superior to the senses dating from
the Greeksat least Plato..thought experiments - Reason
- Logic
- Secondary data
- Empiricism
- If it cannot be held, felt or shaken etc it does
not exist! - In its pure form it fails due to particle physics
etc - Positivism versus interpretivism
- Primary data
15Assumptions - preliminary concepts
- What are the assumptions under which academic
research is conducted? - The world is understandable.
- We are interested in understanding it.
- Researchers will be open minded.
- Researchers are capable of objective reasoning.
- We can find data/evidence to explore the
phenomena and therefore understand them. - The nature of the understanding can be both broad
and also detailed. - Occams razor points out that too much complexity
can obscure what is happening. - We always have to use ceteris paribus all other
things being equal. - That some degree of cause and effect can be
identified/assumed. - We all operate in terms of our cognitive capacity
which changes, and hopefully develops/increases
over time.
16The Issue of Integrity Fleischmann and Pons
17Integrity and Professor Woo Suk Hwang And
Professor Sir Cyril Burt
Sir Cyril Burt Reported nonexistent studies on
I.Q. tests in the 20s and 30s, and added
fictitious collaborators to his papers
18Some ideas to examine
- Relevance ..doing the right thing
- Relevance ..to whom or to what?
theory/academics/practitioners......Is it
significant? - Rigor .doing the thing right
- Is rigor a dimension of validity?
- If the research is not valid then do not worry
about reliability? - If the research is not reliable can it be valid?
- If the research is neither valid or reliable then
is it worthless? - What can go wrong question, data, analysis,
finding conclusions etcthreats to the research
19Evaluation of relevance and rigor
- Relevance is normally evaluated holistically i.e.
is the research important or is it not. Sometimes
some aspects of the research may be more useful
than others but this question is essentially a
holistic one. - Rigor is normally evaluated on an issue by issue
basis. This requires comment of individual
aspects of the research. But there is also a
holistic view. There may be some tension between
the individual evaluations and a holistic view of
the research. The sum of the parts may be
different to the whole.
20Relevance
- In information science Relevance is the extent
to which some information is pertinent,
connected, or applicable to the matter at hand.
It represents a key concept in the fields of
documentation, information science, and
information retrieval. Pehcevski and Larsen - Although academic relevance is today of essential
importance to research, surprising little has
been written about it. - It used to have been assumed that if the research
is good then it is academically relevant and
this attitude has only recently been challenged. - The main drivers of this challenge to relevance
may be seen in the debate surrounding Mode 1 and
Mode 2 research.
21A stumbling block issue
- Previously relevance was not a visible issue. If
the research and his/her supervisor thought that
the question was important then that was all that
was needed. - Now the research has to convince the examiners or
the reviewers that it is adequately relevant. An
argument has to be made. - A research dissertation/paper could be
refused/rejected on the grounds that the
research is not adequately relevant
22Weak and strong relevance
- According to Blake Ives (US IS academic) there
are two forms of relevance - weak relevance and
strong relevance - http//www.misq.org/archivist/v
ol/no20/issue3/edstat.html - Weak relevance occurs when a research question
addresses an issue which is known and documented - Strong relevance requires an action research
approach to answering the research question which
produces concrete findings
23A closer look at relevance
- The statement that a research question is
relevant begs the question of relevant to whom?
and in which way? - A research question is relevant if there is-
- A distinct interest among the stakeholders but
who are the stakeholders? A material stakeholder
group - The research question will provide interesting
results and/or useful guidelines to professionals - Improvements to practice i.e. there is clear
application for the findins
24How are relevant research questions to be
identified?
- Relevant research questions may be found in-
- the extant literature
- work place experiences
- special interest groups
- Business/professional/practitioner groups
- government requests
- general business press
- Remember that a relevant question leads to a
solution for someone - Is there pure or basic research in BM
Studies?
25What do we need to find a relevant research
question?
- Imagination!
- .. But also needed is imagination to create
from these hints the great generalizations to
guess at the wonderful, simple, but very strange
patterns beneath them all, and then to experiment
to check again whether we made the right guess.
Richard Feynman, Six Easy Pieces, p2, Penguin
Books, London (1995) - Finding a suitable research question should not
be rushed. - Activities which can help to find relevant
research questions include- - Talking to knowledgeable informants
- Focus groups
- Qualitative questionnaires
- Blue sky and scenario sessions
- Talking to your supervisor/s
- Literatureacademic and popular press
26A caveat!
- The relevance will be judged by both academics
and by practitioners but academics will/may have
a greater say than practitioners. - The degree of relevance is often tested by the
So What? or Whats new ....... and who cares?
question. This question needs to be directly
answered in a research dissertation and in a peer
reviewed academic paper. - Relevance or the lack thereof will always be
known post hoc or a posteriori but you have to
address this issue on an a priori basis
27The time dimension
- Relevance has a time dimension.
- What is relevant today may not be relevant
tomorrow. - The time value of a research question is a
function of the field of study and the topic and
directly influences the relevance of the research.
28The argument
- Relevance may not be taken for granted.
- The researcher needs to argue that the research
is relevant because it- - answers a well known /important question/s
- addressed a non-obscure question
- will produce a useful result usually a paradigm
- will allows several other questions to be
explored - it will attract people and funding to the
university or business school - etc
29The argument is one of the key issues for
establishing relevance and rigor
30Assessing relevance
- Feedback from informants and peers is the most
effective way of ensuring that the research is
relevant. - It is useful to have in mind the need to test
this relevance occasionally through out the
duration of the research - Also test the findings by asking the stakeholders
of the results of the research are likely to be
useful to them.
31Journal titles......Relevant?
- Stark, S., Chernyshenko, O., Guenole, N.
(Accepted). Can subject matter expert ratings of
statement extremity be used to streamline the
development of unidimensional pairwise preference
scales. (18 no ?) Organizational Research
Methods - Chernyshenko, O., Stark, S., Guenole, N.
(2007). Can the discretionary nature of certain
performance criteria lead to differential
prediction across cultural groups? (15 yes or no
answer) International Journal of Selection and
Assessment, 15, 175-184 - Be careful of papers with very long titles and
many authors.