Open%20assessment%20and%20metal%20risks?

1
Open assessment and metal risks?

• Jouni Tuomisto
• KTL, Finland

2
Yleismalli-wiki
3
Assessment page diagram
4
Assessment page variables
5
Module
6
General model of metal risks
7
Analytica variable and Wiki link
8
Variable scope
9
Variable definition
10
Variable result
11
The attributes of a variable

• Name (an identifier)
• Scope (research question and boundaries)
• Definition
• Data (relevant information)
• Causality (dependencies)
• Unit
• Formula (how to compute based on the above)
• Result (what is the current estimate/answer to
  the research question?)

12
The attributes of a variable
13
Connections between variable attributes

14
Parts of an attribute

• Actual content what is known
• Narrative description any explanations or
  background information that is useful to
  understand the actual content
• Discussion (formal) discussions about the actual
  content

15
What is the acceptability of the idea of open
assessment?

• Poll (informal, based on observations of several
  audiences)
• 30 think it is a stupid idea
• 50 think it cannot work
• 15 find it interesting, but
• 5 are fond of the idea
• In which category do you fall in?

16
Interfaces between tools

• Opasnet Open website for any assessment-related
  descriptions (en.opasnet.org, fi.opasnet.org)
• Erac website A project website for Finmerac
  material (to be extended to other projects)
• Analytica modelling software (Monte Carlo,
  causal diagrams and models. www.lumina.com)
• Models uploaded to Opasnet
• Computing in Analytica, descriptions in Opasnet.
• Result base Interpreted results in a
  model-friendly format (www.pyrkilo.fi/resultdb,
  soon base.opasnet.org)
• M-files For original and processed data (FMS)

17
Lessons learnt

• Finmerac was not originally planned as an open
  assessment. This caused confusion when the way of
  working changed during project.
• Researchers not experienced in OA or wiki
  reluctance to use.
• Difficulties in finding the role and place of own
  work.
• The role was only found when the assessment
  structure became clear enough

18
End
19
What are the challenges of the current risk
assessment?

• Limited area of application
• Lack of flexibility and breadth
• Inefficiency and slowliness of the process
• Deliberate biases towards "safety"
• Communication problems
• Lack of acceptability among stakeholders

20
What is needed from the new assessment?
21
Paradigm shift

• Thomas Kuhn (1922-1996)?
• Science progresses in a regular way until too
  many faults are identified in the current
  paradigm. Then, there is a period of
  extraordinary science, which leads into a shift
  of paradigm

22
Open assessment

• The objectives
• Find solutions to ALL the challenges at the same
  time
• Systematize and "industrialize" the risk
  assessment
• Maintain high scientific quality
• The current situation with open assessment there
  are suggested methods to all challenges listed
  previously
• Many of the suggestions have not been tested in
  practice
• Not everything will probably work
• ?However, there is already a critical mass of
  solutions available so that full-scale testing
  can be started
• ?Further problems should be solved as they appear

23
Assessments in practice

• They look like Wikipedia articles
• They are written in much the same way
• Substance is on the main page
• Research question
• Definition how to find an answer
• Result
• Discussion about the substance is on a separate
  discussion page
• Discussion can be free or structured

24
Open assessment

• The research question for the (pyrkilo) method
• "How can scientific information and value
  judgements be organised for societal
  decision-making in such a way that open
  participation is possible?"
• Full range of development
• a new ontological foundation
• strictly object-oriented approach
• a new structure for information objects
• traditional RA methods for processing
  information, but organised in a more systematic
  way
• tools that enable open collaboration
• data sources that are directly available and
  applicable

25
Highlights of the theory behind

• The basic principles are
• Open participation at all phases of the
  assessment
• Requirement of scientific method (openness to
  scientific critique) at all phases
• Reusability of information from one assessment to
  another
• A uniform information structure is used
• Assessment specific information need for a
  policy decision
• Variable a truthful description of a particular
  part of reality it is independent of assessments
  given its scope.
• Both objects have 4 attributes name, scope
  (research question), definition (how to answer),
  and result

26
Assessments are performed openly in the Internet

• Parts needed to run open assessments
• Web pages with method descriptions for the
  assessors (Guidebook)
• Web pages about useful data (Resource Centre)
• Web pages about actual assessments
• Descriptions of assessments and models used
  (Warehouse)
• Actual models (Toolbox)
• Results of assessments in a uniform structure
  (Result database)
• Web pages for collaborative work on assessments
  (Collaborative workspace)
• Red parts are officially parts of the Intarese
  project

27
Quality control

• Is quality control possible if pages are
  available for all to contribute?
• Our first thoughts (no experience yet, because
  work is practically within research projects)
• Pages can be freely added and edited
• Advanced pages are protected from direct edits
• Then, the edits are made through an open
  discussion on a separate page results are
  transferred to the actual page
• Very important pages go through a peer review and
  get a quality label

28
The ORA report
29
Falsification

• Karl Popper (1902-1994)?
• Science consists of statements (theories) that
  can be falsified
• Science is an evolutionary process where poor
  theories are falsified
• The current knowledge consists of those theories
  that have not (yet) been falsified

30
Argumentation

• Frans van Eemeren
• Disputes can be solved by using formal
  argumentation that consists of attacks and
  defends of specified statements

31
Mass collaboration

• Don Tapscott, Anthony Williams
• A large group of unorganised people are able to
  produce complex artefacts, if the product is
  information or culture, the work can be chopped
  into bite-size pieces, and the pieces can be
  effectively synthesised.

32
Summary

• The purpose of open assessment is to improve and
  make more efficient the use of scientific
  information
• The basic principles are
• Open participation at all phases of the
  assessment
• Requirement of scientific method (openness to
  scientific critique) at all phases
• Reusability of information from one assessment to
  another
• Researchers are needed to test the ideas of open
  assessment
• If successful, scientific practices may change