The Mysteries of the AAPOR Code

1
The Mysteries of the AAPOR Code

• What is it and what happens if someone violates
  it?
• Mary E. Losch Stephen Blumberg
• AAPOR Standards Chair Associate Standards Chair
• Presentation to DC AAPOR
• March 2009

2
AAPOR Code of Professional Ethics and
Practicehttp//www.aapor.org/aaporcodeofethics
Schedule of Proceduresfor Code
Violationshttp//www.aapor.org/scheduleofprocedu
resforcodeviolations
3
I. Principles of Professional Practice in the
Conduct of Our Work

• A. We shall exercise due care in developing
  research designs and survey instruments, and in
  collecting, processing, and analyzing data,
  taking all reasonable steps to assure the
  reliability and validity of results.
• 1. We shall recommend and employ only those
  tools and methods of analysis that, in our
  professional judgment, are well suited to the
  research problem at hand.
• 2. We shall not knowingly select research tools
  and methods of analysis that yield misleading
  conclusions.
• 3. We shall not knowingly make interpretations
  of research results that are inconsistent with
  the data available, nor shall we tacitly permit
  such interpretations.
• 4. We shall not knowingly imply that
  interpretations should be accorded greater
  confidence than the data actually warrant.

4
I. Principles of Professional Practice in the
Conduct of Our Work

• B. We shall describe our methods and findings
  accurately and in appropriate detail in all
  research reports, adhering to the standards for
  minimal disclosure specified in Section III.
• C. If any of our work becomes the subject of a
  formal investigation of an alleged violation of
  this Code, undertaken with the approval of the
  AAPOR Executive Council, we shall provide
  additional information on the survey in such
  detail that a fellow survey practitioner would be
  able to conduct a professional evaluation of the
  survey.

5
II. Principles of Professional Responsibility in
Our Dealings With People

• A. The Public
• 1. When preparing a report for public release we
  shall ensure that the findings are a balanced and
  accurate portrayal of the survey results.
• 2. If we become aware of the appearance in
  public of serious inaccuracies or distortions
  regarding our research, we shall publicly
  disclose what is required to correct these
  inaccuracies or distortions, including, as
  appropriate, a statement to the public media,
  legislative body, regulatory agency, or other
  appropriate group, to which the inaccuracies or
  distortions were presented.
• 3. We shall inform those for whom we conduct
  publicly released surveys that AAPOR standards
  require members to release minimal information
  about such surveys, and we shall make all
  reasonable efforts to encourage clients to
  subscribe to our standards of minimal disclosure
  for their releases.

6
II. Principles of Professional Responsibility in
Our Dealings With People

• B. Clients or Sponsors
• 1. When undertaking work for a private client,
  we shall hold confidential all proprietary
  information obtained about the client and about
  the conduct and findings of the research
  undertaken for the client, except when the
  dissemination of the information is expressly
  authorized by the client, or when disclosure
  becomes necessary under the terms of Section I-C
  or II-A of this Code.
• 2. We shall be mindful of the limitations of our
  techniques and capabilities and shall accept only
  those research assignments that we can reasonably
  expect to accomplish within these limitations.

7
II. Principles of Professional Responsibility in
Our Dealings With People

•  C. The Profession
• 1. We recognize our responsibility to the
  science of survey research to disseminate as
  freely as possible the ideas and findings that
  emerge from our research.
• 2. We shall not cite our membership in the
  Association as evidence of professional
  competence, since the Association does not so
  certify any persons or organizations.

8
II. Principles of Professional Responsibility in
Our Dealings With People

• D. The Respondent
• 1. We shall avoid practices or methods that may
  harm, humiliate, or seriously mislead survey
  respondents.
• 2. We shall respect respondents' concerns about
  their privacy.
• 3. Aside from the decennial census and a few
  other surveys, participation in surveys is
  voluntary. We shall provide all persons selected
  for inclusion with a description of the survey
  sufficient to permit them to make an informed and
  free decision about participation.

9
II. Principles of Professional Responsibility in
Our Dealings With People

• The Respondent
• 4. We shall not misrepresent our research or
  conduct other activities (such as sales, fund
  raising, or political campaigning) under the
  guise of conducting research.
• 5. Unless the respondent waives confidentiality
  for specified uses, we shall hold as privileged
  and confidential all information that might
  identify a respondent with his or her responses.
  We also shall not disclose or use the names of
  respondents for non-research purposes unless the
  respondents grant us permission to do so.
• 6. We understand that the use of our survey
  results in a legal proceeding does not relieve us
  of our ethical obligation to keep confidential
  all respondent identifiable information or lessen
  the importance of respondent anonymity.

10
III. Standards for Minimal Disclosure

• Good professional practice imposes the obligation
  upon all public opinion researchers to include,
  in any report of research results, or to make
  available when that report is released, certain
  essential information about how the research was
  conducted. At a minimum, the following items
  should be disclosed.
• 1. Who sponsored the survey, and who conducted
  it.
• 2. The exact wording of questions asked,
  including the text of any preceding instruction
  or explanation to the interviewer or respondents
  that might reasonably be expected to affect the
  response.
• 3. A definition of the population under study,
  and a description of the sampling frame used to
  identify this population.

11
III. Standards for Minimal Disclosure

• 4. A description of the sample design, giving a
  clear indication of the method by which the
  respondents were selected by the researcher, or
  whether the respondents were entirely
  self-selected.
• 5. Sample sizes and, where appropriate,
  eligibility criteria, screening procedures, and
  response rates computed according to AAPOR
  Standard Definitions. At a minimum, a summary of
  disposition of sample cases should be provided so
  that response rates could be computed.
• 6. A discussion of the precision of the findings,
  including estimates of sampling error, and a
  description of any weighting or estimating
  procedures used.
• 7. Which results are based on parts of the
  sample, rather than on the total sample, and the
  size of such parts.
• 8. Method, location, and dates of data collection.

12
What happens if there is a violation?
13
Schedule of ProceduresLast Revised -- January
2006

• 1. Any individual, whether an AAPOR member or
  not, may submit a written complaint concerning a
  possible violation of the AAPOR Code of
  Professional Ethics and Practices to the
  Standards Chair.
• 1a. Complaints must be made in writing on an
  AAPOR Standards Complaint Form and must
  include
• (i) a statement describing the alleged action or
  violation,
• (ii) the section of the Code alleged to be
  violated,
• (iii) the date or dates when the alleged
  violation occurred,
• (iv) the name(s) of the individual(s), firm(s) or
  organization(s) alleged to be in violation, and
• (v) the original signature of the complainant
  along with his or her contact information,
  including telephone number, postal address, and
  e-mail address.

14
Schedule of Procedures

• 1b. Any organization, firm, individual, or group
  of individuals may be named in such a complaint,
  whether or not it or they are members of AAPOR.
• 1c. The written complaint shall be accompanied
  by all available relevant evidence.
• 1d. At the discretion of the Standards Chair,
  any duly appointed member of the Standards
  Committee may be given confidential access to a
  received complaint, unless that Standards
  Committee member, or his or her organization, is
  the source or subject of that complaint. If the
  Standards Chair and Associate Standards Chair are
  thereby excluded, the Council shall appoint a
  Chair pro tem for that particular complaint.

15
Schedule of Procedures

• 1e. The complaint form and all related materials
  prepared for Council, the Standards Committee,
  and the Evaluation Committee (if one is appointed
  as provided in section 4 below) will be kept
  confidential, and all Council, Standards
  Committee, and Evaluation Committee discussions
  related to the specifics of the complaint will be
  treated as privileged communication.
• 1f. The procedures hereafter described will
  apply to all such written complaints received by
  the Standards Chair.

16
Schedule of Procedures

• 2. Within forty-five (45) days of receipt of a
  complaint, the Standards Chair shall decide, with
  the concurrence of the Associate Standards Chair
  or one other member of the Standards Committee,
  whether or not any official action is warranted.
  The Standards Chair may request assistance from
  the source or subject of the initial complaint,
  or use any other channels deemed helpful in
  recommending a course of action to Council.

17
Schedule of Procedures

• 2a. If no official action is to be recommended
  by the Standards Chair, he or she shall prepare a
  brief written report to Council, including the
  original complaint form and setting out the
  reason(s) why no action is indicated.
• (1) If Council agrees with this opinion at
  its next regularly scheduled meeting, or after
  being polled by phone or e-mail, the source of
  the complaint shall be promptly notified as to
  the final disposition of the complaint.
• (2) If Council does not agree with the
  Standards Chairs decision, Council may call for
  clarification or direct the initiation of a full
  complaint review and investigation, as specified
  in sections 2b, 2c, and 3 below.

18
Schedule of Procedures

• 2b. The Standards Chair may be directed by
  Council to seek additional information and
  clarification about the alleged Code violation.
  In pursuit of this objective, the Standards Chair
  may request assistance from the source or subject
  of the initial complaint, or use any other
  channels deemed helpful in recommending a course
  of action to Council. The Standards Chair shall
  report on any additional information within
  thirty days of such a directive from Council.
• 2c. The Standards Chair may recommend and a
  Council majority approve of (or independently
  direct) the initiation of a full complaint review
  and investigation. In determining this course of
  action, Council will consider all relevant
  information, including the severity of the
  alleged violation(s) and facts gathered by the
  Standards Chair, including (if elicited) the
  stance initially taken by the subject(s) of the
  complaint.

19
Schedule of Procedures

• 3. If Council, by a majority of those voting,
  votes for a complaint review and investigation,
  the organization, firm or individual(s) who are
  the subject of the complaint shall be notified by
  the Standards Chair in writing within fourteen
  (14) days of Council action. Such notification
  will fully stipulate the nature of the complaint
  and shall include relevant portions of the Code
  and a copy of this Schedule of Procedures.

20
Schedule of Procedures

• 4. The investigation will be conducted by an ad
  hoc committee (hereafter termed Evaluation
  Committee), appointed by the Standards Chair
  subject to the approval of the Council.
• 4a. The Evaluation Committee will consist of
  no fewer than three nor more than five AAPOR
  members, one of whom Council will name as Chair.
  Neither the Standards Chair nor any other Council
  members may serve on this committee.
• 4b. The identity of the members of the
  Evaluation Committee will be known only to
  Council and to each other.
• 4c. No member of AAPOR shall serve on an
  Evaluation Committee in a matter where he or she
  or his or her organization is the source or
  subject of a particular complaint, or where his
  or her service could otherwise represent a
  potential conflict of interest in relation to the
  source or subject.

21
Schedule of Procedures

• 5. Within twenty-one (21) days of appointment,
  the Chair of the Evaluation Committee will
  receive all pertinent materials on the case that
  are held by the Standards Committee, and the
  Standards Chair will, in writing, request the
  subject(s) of the complaint to provide any other
  materials deemed necessary by the Evaluation
  Committee, or materials and statements which the
  subject(s) deem necessary or relevant to a fair
  investigation.

22
Schedule of Procedures

• 5a. Copies of all such materials received,
  including the original complaint and the
  notification to the subject(s), will be sent to
  each member of the Evaluation Committee.
• 5b. Any member of the Evaluation Committee may
  request the Standards Chair to seek further
  information or clarification of existing
  information.
• 5c. In conducting its affairs, the use of
  electronic conferencing techniques (phone,
  e-mail) by the Evaluation Committee, as needed,
  will be funded by AAPOR.
• 5d. The Committee may elicit and consider any
  relevant evidence.

23
Schedule of Procedures

• 6. If the subject(s) of the complaint fails to
  respond to the notice provided under paragraph 3
  or fails to forward all material requested under
  paragraph 5 or 5b within forty-five (45) days of
  the notice or request, the Evaluation Committee
  will proceed, using the evidence at hand. If,
  however, the subject(s) of the complaint formally
  requests of the Standards Chair an extension of
  time, the Standards Chair may grant an extension
  in writing, upon determination that the extension
  is reasonable and necessary to compile all
  requested materials.

24
Schedule of Procedures

• 7. Within thirty (30) days of receiving the
  requested materials (or following the 45-day
  waiting period), the Evaluation Committee will
  complete a complaint review and investigation,
  reach a determination about the complaint, and
  will submit a written report to the Standards
  Chair. The complaint review and investigation
  will seek to determine the precise nature of the
  conduct which is subject to criticism and whether
  there is any factual basis to support the alleged
  misconduct. The Evaluation Committee must
  determine whether the information obtained may
  reasonably be interpreted to constitute a
  violation of AAPORs Code of Ethics and
  Practices, which reflects generally accepted
  professional practices. The committees report
  will describe the steps taken in review of the
  complaint, include the determination and a
  specific recommendation for Council action, and
  will conclude with one or more motions.

25
Schedule of Procedures

• 7a. Council action to be recommended by the
  Evaluation Committee may consist of, but is not
  limited to, any of the following
• (1) If AAPOR members or firms with AAPOR members
  as principals are subjects of the complaint
• a. Public or private exoneration
• b. Public or private censure
• c. Suspension or termination of membership
• d. Termination of investigation, with no
  further action.
• (2) If non-members or firms without AAPOR
  members as principals are subjects of the
  complaint
• a. Public or private exoneration
• b. Public or private censure
• c. Termination of investigation, with no
  further action.

26
Schedule of Procedures

• 7b. Before recommending any censure or altering
  of membership, the Committee should consider the
  following criteria
• (1) The degree of severity of the violation(s)
• (2) The subjects history regarding other
  possible violations
• (3) The impact of the violation(s) on the
  public opinion and survey research profession
• (4) The impact of any proposed sanction(s) on
  the subjects ability to successfully participate
  in public opinion and survey research in the
  future.
• 7c. Notwithstanding other actions taken, the
  Council may direct that a case study be prepared
  and publicized for the benefit of AAPOR members
  and other constituencies, with due regard given
  to the need to maintain confidentiality of the
  subject(s), information sources, and methods of
  investigation.

27
Schedule of Procedures

• 8a. The Standards Chair will forward the
  Evaluation Committees report to Council within
  fourteen (14) days of receipt, together with such
  views, in writing, as he or she may have. At its
  next regularly scheduled meeting thereafter
• (1) The Council (with at least two-thirds of its
  members voting) shall immediately move to accept,
  reject, or modify the report of the Evaluation
  Committee or of the Standards Chair.

28
Schedule of Procedures

• 8a1 a. If the Councils decision exonerates the
  subject of the complaint, the Standards Chair
  will promptly notify the subject(s) and no
  further Council action is required.
• b. If the Councils decision does not exonerate
  the subject, the Standards Chair will send notice
  of the Councils decision to the subject by
  registered mail within fourteen (14) days. The
  Standards Chairs letter shall include written
  notification advising the subject
• (i) of the specific allegations and charges
• (ii) that, within thirty (30) days of receipt
  he or she may request a hearing before the full
  Council and
• (iii) that he or she will be given the right
  at said hearing to confront evidence and to
  refute all charges.

29
Schedule of Procedures

• 8a1c. The Council may also decide to request
  modification to the report of the Evaluation
  Committee. If such a request is made by Council,
  the Evaluation Committee has thirty (30) days to
  respond to such request. The revised report is
  then resubmitted for consideration, following the
  steps outlined in 8.1.

30
Schedule of Procedures

• 8a2 Within thirty (30) days of receipt of
  notice of Councils decision (section 8a1b),
  the subject of the complaint may request, in
  writing, a hearing before the full Council. Such
  a hearing shall be held at a time and place
  scheduled by the Council. The President shall
  control the procedure at the hearing, including
  presentation of testimony and submission of
  written materials, but shall not be bound by
  legal rules of evidence or procedures. Before the
  close of the hearing, the Council should consider
  the criteria in 7b above.

31
Schedule of Procedures

• 8b. Immediately following the close of the
  hearing - or, if no hearing is requested within
  the 30-day waiting period, at a regularly or
  specially convened meeting - the Council will
  make its decision (with at least two-thirds of
  its members voting), to accept, reject, or modify
  the judgment and recommendation of the Evaluation
  Committee or the Standards Chair.
• 8c. This decision will constitute the final
  action to be taken by the Council on behalf of
  AAPOR, except as allowed in section 7c. The
  subject of the complaint and the source of the
  original complaint will be notified in writing of
  the Councils decision within fourteen (14) days
  by the President of AAPOR, who also will be
  responsible for implementing the terms of the
  decision.

32
The recent public case

• An allegation of violation of code was received
  in March 2008
• Following the procedures, a review by Standards
  Council resulted in a formal investigation by an
  evaluation committee
• The committee reviewed the evidence, made formal
  requests to the subject of the complaint for
  elements of minimal disclosure

33
The specific requests related to AAPORs finding
of violation of minimum disclosure were as
follows

• The survey sponsor(s) and sources of funding for
  the survey.
• A copy of the original questionnaire or survey
  script used in the 2006 survey, in all languages
  into which it was translated.
• The consent statement or explanation of the
  survey purpose.
• A full description of the sample selection
  process, including any written instructions or
  materials from interviewer training about sample
  selection procedures.
• A summary of the disposition of all sample cases.

34
The specific requests related to AAPORs finding
of violation of minimum disclosure were as
follows

• How were streets selected? How were the starting
  street, and the starting household, selected?
  Once the starting point was selected, how were
  interviewers instructed to proceed (e.g., when
  they came to an intersection)? How were houses
  and respondents chosen at housing units?
• The survey description says that, The interview
  team were given the responsibility and authority
  to change to an alternate location if they
  perceived the level of insecurity or risk to be
  unacceptable. In how many clusters did the team
  change location, and what were the reasons for
  the changes?

35
The specific requests related to AAPORs finding
of violation of minimum disclosure were as
follows

• The survey description says that, Empty houses
  or those that refused to participate were passed
  over until 40 households had been interviewed in
  all locations. Were such cases included in the
  number of not-at-home and refusal cases counted
  in each cluster?

36
Dr. Burnhams response was as follows

• This study was carried out using standard
  demographic and household survey methods.
• The methods we employed for this study were set
  out in the Lancet paper reporting our findings
  (Lancet, 20063681421-28). The dataset from the
  study was released some time ago.

37
The Outcome

• In accordance with AAPORs procedures, the
  evaluation committee completed its work and
  recommended public censure which was accepted by
  Council
• The public censure was manifested in the press
  release of February 4th, 2009
• Then all hell broke loose -)

38
Reactions

• The media especially international were
  interested in the action
• Some bloggers began flogging You are not the
  boss of him
• ASA was notably supportive
• A number of AAPOR members were delighted
• A number of AAPOR members seemed convinced that
  this Council was out of control and following no
  known guidelines See second bullet above You
  are not the survey police

39
Updates

• Dr. Burnham has not publicly responded to the
  AAPOR censure to our knowledge has stated only
  that JHU instructed him not to respond
• JHU recently announced that the IRB had
  sanctioned Dr. Burnham for the same study they
  have suspended Dr. Burnham's privileges to serve
  as a principal investigator on projects involving
  human subjects research for 5 years
• The specific details of the minimal disclosure
  request from AAPOR to Dr. Burnham (provided
  earlier in the slides) is now posted on the AAPOR
  website

40
Questions?