Web-Based Research: Issues, Problems and an Example Technique

1
Web-Based ResearchIssues, Problems and an
Example Technique

• John H. Krantz
• Hanover College

2
Outline

• Brief History
• Issues in Internet Research
• An Example Method
• Problems and Solutions

3
History

• Email for 2-3 Decades
• 1995
• Krantz, Ballard, Scher (1997) Within Subjects
• Reips (1997) Between Subjects
• First True Web Experiments
• July 1995 List of Online Psychological Research
  sponsored by APS for awhile
• http//psych.hanover.edu/research/exponnet.html
• Almost 200 studies and study sites listed

4
Why Do Internet Research

• Musch Reips (2000)
• Sample Size
• Statistical Power
• High Speed
• Ability to reach participants in other countries
• High external or ecological validity
• Low cost
• Ability to replicate a lab experiment with more
  power
• Special populations

5
Issues

• Is the Sample Representative
• Yes and No
• Important to remember base of comparison
• Ideal vs. Lab
• Krantz Dalal (2000)
• Much more diverse that most lab samples
• Especially in age and education range
• Race is generally limited as is nationality

6
Sample Characteristics Gender

• Are Internet Samples Male Dominated?
• Krantz Dalal (2000) Female
• GVU 1st (1994) 5
• Reips (1996) English 43 German 18
• Krantz, et al. (1997) 44
• Pasveer Ellard (1998) 3rd Study 71
• More Recent
• Caddell Utt (2004) 77
• Meyerson Tryon (2003) 45

7
Sample Characteristics Age

• Are we still testing college sophomores?
• Krantz, et al. (1997) 43 gt 30
• Smith Leigh (1997) 35 gt 30
• Pasveer Ellard (1998) 45 gt 25
• GVU 1st 36 gt 30
• Caddell Utt (2004) 60 gt 30
• Pattison Rouse(2003) 16 gt 30

8
Is the College Sophomore be Making a Comeback

• Plot of activity of Psychological Research on the
  Net
• Pattison Rouse (2004) 76 18-22

9
Sample Characteristics Race

• How diverse are the samples?
• Unfortunately in general samples, diversity is
  still limited
• Krantz et al. (1997) 89 White
• Smith Leigh (1997) 86 White
• GVU 10th (1998b) 87 White
• ONeil, Penrod, Bornstein (2003) 82 White
• Meyerson Tryon (2003) 93 White
• However, as will be discussed later, web can make
  it possible to access special populations

10
Sample Characteristics Nationality

• Where to do the subjects come from?
• Largely North American, even US, even in some
  European studies
• Krantz, et al. (1997) 86 N. Am.
• Senior, et al (1999) gt80 N. AmThis study was
  conducted in England.

11
Is the Data Any Good

• Emphatically Yes

12
Data Quality Direct Comparisons

• A number of studies, still, run both laboratory
  and internet samples.
• Krantz, et al. (1997)
• Regression of web means on laboratory samples
  (even though different types of samples) lab
  mean 1(Web mean) 0 r2 .99

13
Compare to Established Data

• Compare web results to previously published data
  sets
• Myerson Tryon (2003)
• Studied Sexual Boredom Scale of Watt Ewing
  (1996)
• Matched sample characteristics
• Found same internal consistency
• Form of administration was not a significant
  factor

14
Direct Validity Comparison

• Use same techniques to validate results
• Pasveer Ellard (1998)
• Developed new scale
• Internal consistency
• Psychomectric properties

15
Data Validity Arguments

• Reips (2000)
• Statistical Power
• Limited Sample Population
• Limited External Validity
• Less than Optimal Voluntariness
• Motivational Confounding
• Experimenter Bias
• Nontransparency
• Limitations of what is feasible to research

16
Interesting Deviations

• Stern Faber (1997)
• Milgrams lost letter technique
• Milgram, sent on
• Stern Faber, returned to sender
• From doing favor to doing easy task
• Often effects are smaller

17
Getting Subjects General

• http//psych.hanover.edu/research/exponnet.html
• Other Pages
• Social Psychology http//www.socialpsychology.org
  /expts.htm
• The Web Experiment List http//genpsylab-wexlist.
  unizh.ch/

18
Getting Subjects Special Populations

• Advertising
• Email Groups
• Careful, get permission
• Can be thought spam
• Netiquette
• Discussion Groups
• Same, permission

19
Sample Method for Doing Survey

• Birnbaum, M. H. (2001). Introduction to
  Behavioral Research on the Internet. Upper
  Saddle River, NJ Prentice Hall.
• Lots of resources at http//ati.fullerton.edu/
• NSF sponsored classes at Fullerton on Social
  Psychology
• One more next January, look for the announcement
  at this site

20
First Step Where to put it

• Setting up a server can be easy
• Apache web server
• http//www.apache.org
• Most popular server
• Freeware
• Any computer connected to web is now a server
• May have issues of firewalls dynamic IP
  addresses

21
Second Step SurveyWiz

• In Birnbaum (2001)
• Also linked to from http//ati.fullerton.edu/
• Under Birnbaums page
• Freely available
• We will build a very short survey to illustrate

22
Forms

• SurveyWiz uses web forms to collect data
• What are forms
• Elements to allow web page viewers to input
• This data is then sent back to the server to be
  processed
• SurveyWiz
• What the page looks like My Take
• Can edit it with any web page editor

23
Examples I have done this year

• Caddell Utt
• Pattison Reese

24
Third Step Collecting the data

• First way, let Birnbaum do it for you
• SurveyWiz is set up this way
• ltFORM actionhttp//psych.fullerton.edu/cgi-win/po
  lyform.exe/generic methodpostgt
• FTP//guestguest99_at_psych.fullerton.edu
• You can type in the address box of your browser.
  Note that this is FTP site. At the present time,
  password is guest99, as shown above, and permits
  download only.

25
Keeping the Data Local

• Second, get this perl script (found at
  http//psych.fullerton.edu/mbirnbaum/programs/PERL
  _script2.htm)
• The script and instructions are at this site
• Written by Billy Schmidt
• Also change code in form command
• At my school the line reads
• ltFORM actionhttp//psych.hanover.edu/cgi-bin/surv
  ey_wiz.pl methodpostgt

26
What is CGI

• Common Gateway Interface
• Method for other programs to interact with web
  servers
• In this case, this perl program takes data from
  web forms and stores them in a file
• Stores data in Excel or SPSS readable files

27
Fourth Step Example Data File

• Data File
• Data coding scheme all variable names begin two
  numbers, see example
• The perl program sorts the variables by numerical
  order

28
Another Method of Survey Generation

• Schmits Survey Assistant
• He manages data
• More flexibility
• http//or.psychology.dal.ca/wcs/hidden/home.html
  

29
What Do I Need?

• At NSF ATI site
• List of software resources

30
What Problems are important

• Musch Reips (2000) 5 point scale
• No control over participants behavior 3.6
• No control over motivation 3.4
• Inability of participants to ask questions 3.3
• Nonrepresentative sample 2.9
• Manipulation and fraud 2.4
• Ethical problems 1.5

31
Problems

• Eliminating Multiple Entries
• Dropout
• Security
• Data Integrity

32
Multiple Entries

• People submit too fast, while waiting for
  feedback
• People will fake being two people
• Usually have same IP address
• SurveyWiz and most other methods, sends the IP
  address of machine where survey is being taken
• Can eliminate more than one from same IP, e.g.
  Schmidt (1997)

33
Security Data Integrity

• If on public server, others can access data and
  download
• Others might fake pages to send data (unlikely)
• Keep data in non-public directories so only
  researchers have access
• Have CGI check for origin of survey to make sure
  it is yours (Schmidt, 1997)

34
Dropout

• People will come but not finish
• Or data is incomplete
• Survey will sends a complete signal to help you
  track
• Can use (ONeil, Penrod, Bornstein, 2003)
• Short sweet
• Financial incentives.
• If multiple pages to survey, several pages
  warm-up so not drop-out during

35
Ethical Issues

• Should not think of Ethical Issues of Web
  Research Alone
• Compare to Ethics of doing Traditional Research
• Not let our comfort with what we have done blind
  us to those problems

36
Conclusion

• Many Benefits
• Easy Methods
• Easy Resources
• Not alternative to traditional methods
• A new tool still needs to be used thoughtfully