Cost effective use of simulations in online assessment

1
Cost effective use of simulations in online
assessment

• Ruth Thomas, Colin Milligan, Bill Austin
• JeLSIM Partnership
• Helen Ashton, Cliff Beevers, Dave Edwards
• Heriot-Watt University

2
Simulations and assessment

• Greater range of questions
• Greater interactivity
• Can test higher order skills
• More authenticity
• Feedback
• Intrinsic

3
Cost effectiveness

• Keep costs low.
• Re-use pre-existing simulations
• Multi-purpose new simulations
• Avoid use of programmers
• Keep effectiveness high.
• Understanding of range of uses
• Staff confidence and empowerment
• Simulation developers should consider assessment.

4
The aim of the presentation

• Look at the technological perspective
• Develop a framework for assessing the potential
  of simulations in assessment
• Practicalities of simulations and assessment

5
(No Transcript)
6
Integration of simulation and AE

1. No integration The simulation can be included in
   a question as a media object, but there is no
   communication between simulation and assessment
   engine.
2. Low level of integration The simulation can pass
   a single variable, answer to the assessment
   engine. There is no facility to set up questions.
   
3. High levels of integration There is two-way
   communication between the AE and simulation to
   set up and mark questions

7
PASS-IT AE-JeLSIM system
8
Educational use of simulations

• Understanding a system
• Developing thinking and problem solving skills
• Learning about processes and procedures
• Modelling

9
Assessments based on understanding the system Assessments based on understanding the system Assessments based on understanding the system Assessments based on understanding the system
Activity Description Example Possible Mechanisms
Relationships between variables If variable X is increased what happens to variable Y? If inflation increases, what happens to unemployment Multiple choice, Free text entry,
Cause and effect If condition A is changed what will the outcome be? What will happen to concentration over time if the initial concentration is changed? Multiple choice, Free text entry, Sketch on a graph
Cause and effect Discover relationships Establish the genetic laws of inheritance for a virtual animal Free text entry Predict result Simulation variables ? AE
Cause and effect What is the value of the missing variable? Examine the behaviour of the sun to determine your latitude. Multiple choice, Numeric input AE, Numeric input sim.
Appropriate ranges Give typical values for variable Z? What is a typical level for background radiation? Multiple choice, Numeric input AE, Numeric input sim.
Conditions and phenomena What combination of variables leads to phenomenon H? Under what conditions will a vehicle skid? Free text Score ? AE Simulation variables ? to AE
Conditions and phenomena Set the simulation to condition J. Set a system of vibrating masses to resonance Score ? AE Simulation variables ? to AE
Conditions and phenomena At what value of variable X does condition Y occur? At what pH will a precipitate begin to appear? Multiple choice, Numeric input AE, Numeric input sim
10
Assessment based on development of thinking skills Assessment based on development of thinking skills Assessment based on development of thinking skills Assessment based on development of thinking skills
Activity Examples Specific Example Possible Answer Mechanisms
Trouble shooting/ problem solving Put the simulation in a state where it functions in a specified way. Keep an endangered population from extinction Scored in simulation score passed to AE for marking. Combination of simulation variables passed to AE
Case-based scenarios Evaluate the viability of a simulated scenario Given a cash flow and trading forecast, comment on the viability of a business. Free text entry. Combination of simulation variables passed to AE
Fault finding Locate the broken components in a system. Which resistors are malfunctioning in a circuit? Scored in simulation score passed to AE for marking Combination of simulation variables passed to AE
11
Assessment based on use of procedures and development of skills Assessment based on use of procedures and development of skills Assessment based on use of procedures and development of skills Assessment based on use of procedures and development of skills
Activity Description Examples Possible mechanisms
Following procedure Follow correct protocols to complete a task Follow emergency resuscitation procedure.   Shutting down a nuclear power station Record variables as task proceeds. Single variable out (score) Multiple variables out (simulation state).
Testing a skill Demonstrate a skill Landing a plane   Single variable out (score) Multiple variables out (simulation state).
12
DC Circuit
Question type Examples
Relationships between variables If voltage increases what happens to current. What effect does changing a resistor have?
Cause and effect If the resistor is changes what will be the effect on a graph of current vs. voltage for a specific resistor
Cause and effect Given a circuit with a number of resistors work out the value of some of those resistors.
Conditions and phenomena What value of resistor X gives a voltage of Y across it?
Trouble shooting/ problem solving If a component requires a certain voltage to operate, given a set of resistors and voltage sources, which combinations allow the component to operate?
Fault finding Determine which resistors are short or open circuit.
Following procedure Follow a procedure to determine the location of a fault.
13
Bizstartup
Question type Examples
Relationships between variables As overheads increase what happens to the break-even point?
Cause and effect If sales are 5 below target what will happen to the bank balance
Cause and effect Some figures are missing from the trading forecast work out what they should be .
Appropriate ranges of variables What are typical costs for various overheads in the simulated business?
Conditions and phenomena Adjust the selling price to achieve a specified breakeven point.
Trouble shooting/ problem solving A business is projected to make a loss. What can be done to bring it into profit?
Case based scenario Given a cash flow and trading forecast consider the viability of the business and the validity of the assumption made
Fault finding There is an error in the trading forecast locate it
Following procedure Given data for a start up company, generate a cash flow and trading forecast
14
Practicalities

• Not all simulation-assessment systems can cope
  with all question types
• Most can use many of less complex types
• Reuse old simulations if possible
• Take care in design of new simulations
• Consider other educational uses
• Maximise re-use from by
• Pedagogic neutrality
• More complex model

15
Aiding question setters

• New technique
• Question setters need confidence
• Examples and framework
• Templates mini question types
• a JeLSIM model
• a template JeLSIM interface
• a question set in PAE to demonstrate
  communications

16
Contact and further info.

• Ruth Thomas rct_at_jelsim.org
• Helen Ashton - H.S.Ashton_at_hw.ac.uk
• JeLSIM website www.jelsim.org
• Exemplars - http//www.calm.ac.uk/sim-asses.html

17
Original applet is at http//www.ida.liu.se/her/
npp/demo.html
18
Original applet is at http//www.ida.liu.se/her/
npp/demo.html
Back