Title: Diapositive 1
1EARLI 11th Biennial conference 2005 electronic
Construction and Quality Control in Standardized
Testing the e-CQCST project Jean-Luc Gilles,
Sylvie-Anne Piette, Pascal Detroz, Salvatore
Tinnirello, Marc Pirson, Mbassa Dabo Hung
Lê http//www.smart.ulg.ac.be/earli-2005/e-CQCST.
ppt University of Liège - Belgium
2I. Context
- Many European Countries have to face a growing
number of higher education students - The training professionals and the human
resources managers of companies are often
constrained to evaluate staff competencies, often
acquired outside the schools - There is a worldwide movement of the introduction
of quality management in education and training
activities
(Kaufman al., 1995 Segers Dochy, 1996
Ramsden, 1991 Nightingale ONeil, 1994 Zink
Schmidt, 1995 Castaigne, Gilles Hansen, 2001
Gilles, 2002 Gilles et Lovinfosse, 2004 Phelps,
2005)
3I. Context
- The quality approach suggested by the SMART team
in this context
4II. Principals RD objectives of the e-CQCST
project
- To develop a web based platform which propose
methods, procedures and tools for the management
of standardized tests. - To set up economic valorizations of the results
of our research with supply of services like - Audit of standardized testing systems
- Training and coaching of the users of the
e-CQCST platform - Docimological services for the improvement of
standardized tests - Test creation with the platform
- To propose a quality label for standardized
testing
5III. Method the CCQCST model
First step 1. ANALYSIS - list of the Points to
Evaluate (PE) - their Priorities (P) - and the
Categories of Performances (CP) of the learners
or trainees End product a test
specification table where Points to Evaluate
(PE) are crossed with Categories of Performances
(CP) ? binomials (PE x CP).
6III. Method the CCQCST model
Second step 2. DESIGN The test developer
must design the Modalities of the Questions (MQ)
according to the Categories of Performances (CP)
previously defined at step 1. End products -
Binomials (CP x MQ) deriving from the Categories
of Performances (CP) associated with the
Modalities of the Questions (MQ) - Trinomials (PE
x CP x MQ) deriving from the binomials (PE x CP),
previously defined at step 1, assembled with
the binomials (CP x MQ) of this step 2 -
Selection of the trinomials which should be
linked to the questions of the test
7III. Method the CCQCST model
Third step 3. QUESTIONS The test developer
selects or writes the questions. These questions
are linked to the trinomials objects selected for
the test at the former step. End product A
series of questions related to the content and
the performances to be test. These questions are
created or selected in a secured item bank
database.
8III. Method the CCQCST model
3. Questions
Selection or writing of the questions which are
linked to the trinomials selected
Item bank
Fourth step 4. TRAINING The person in
charge of the test can choose to propose
trainings on the modalities of the questions
previously defined. End product blank tests
(paper-pencil or web based)
9III. Method the CCQCST model
3. Questions
Selection or writing of the questions which are
linked to the trinomials selected
Item bank
Fifth step 5. TESTING The questions created
or selected at the third step are put in the
standardized test which is proposed to the people
who perform the test. End products - online
exams (in a secured environment)- questionnaires
linked with OMR
4. Training
on the modalities of the questions defined
Blank tests (paper or web based)
10III. Method the CCQCST model
Sixth step 6. CORRECTION The standardized
test developer carries out the correction of the
tests. End products Treatments are designed
to check the quality of the questions before the
final version of the results
3. Questions
Selection or writing of the questions which are
linked to the trinomials selected
Item bank
4. Training
on the modalities of the questions defined
Blank tests (paper or web based)
5. Testing
The standardized test is proposed
Online exams, paper tests linked with OMR
11III. Method the CCQCST model
Seventh step 7. FEEDBACK Personalized and
tailored feedbacks are proposed on the basis of
the final version of the results. End products
Personalized and tailored feedbacks with online
access.
3. Questions
Selection or writing of the questions which are
linked to the trinomials selected
Item bank
4. Training
on the modalities of the questions defined
Blank tests (paper or web based)
5. Testing
The standardized test is proposed
Online exams, paper tests linked with OMR
12III. Method the CCQCST model
Eight step 8. REGULATION the test developer
collects and analyses the opinions of the all the
people who intervened in the test construction
process. End products Advices in order to
improve the next standardized test that will be
created.
3. Questions
Selection or writing of the questions which are
linked to the trinomials selected
Item bank
4. Training
on the modalities of the questions defined
Blank tests (paper or web based)
7. Feedback
5. Testing
The standardized test is proposed
Personalized and tailored feedbacks are proposed
on the basis of final version of the results
Online exams, paper tests linked with OMR
online access FB
13III. Method the CCQCST model
14III. Method the CCQCST model
15V. Companies organizations collaborating in
this project
M. Bernard Van ParysM. Hubert de Broqueville
M. Francis LajoieM. Jean-Pierre Renault
M. Alain Kock
M. Dominique PierretM. Jean-Marie Pairoux
Mme Régine GendarmeMme Catherine DeruisseauMme
Rameault
Mme Sandra SchillemansMme Tania DelhaiseM.
Pierre Ramelot
Mme Maryse RiffontM. Peter Vanderheyden
Mme Catherine WarnantM. Jean-Philippe Léonard
M. Yvan Lagage
16VI. First results Relevance of the CCQCST model
17VI. First results Relevance of the CCQCST model
- Interviews of tests developers of the ten
companies and organizations - Description of the actions
- Collaborators concerned
- Temporal dimension of the tests construction
process - Other processes in which tests are inserted
- Importance of the quality of the standardized
tests. -
- First results of the interviews and the immediate
subcontracting services - The CCQCST model and its procedures are
considered as relevant and well accepted - The principal readjustment we have to plan
consists of an integration of different roles of
tests developers working in teams.
18VI. First results Users roles within e-CQCST
platform
- Two difficulties for the users roles definition
- They must be used in very different companies or
non academic organizations contexts - They should be assigned by one person in charge
of the test (the head of the project) to a series
of collaborators or subcontractors. - We propose seven roles within the e-CQCST
platform - Role 1 INSTALLER ? installs the software on
the server, - Role 2 ADMINISTRATOR ? defines the different
tests supervisors, - Role 3 TEST SUPERVISOR ? defines and
supervises his collaborators
subcontractors for the tasks related to
the test construction, - Role 4 TEST DEVELOPER ? realizes one or more
tasks of one or more steps
of the CCQCST model, - Role 5 QUALITY SUPERVISOR ? defines and
supervises his collaborators or
subcontractors in charge of the
quality controls, - Role 6 QUALITY CONTROLLER ? realizes one or
more tasks related to the
quality controls of the productions of the
tests developers, - Role 7 ASSESSEE ? performs the standardized
test.
19VI. First results The SAFES project
- Being given the success of the subcontracting
services requested by four companies, we filed in
a FIRST SPIN-OFF project of the DGTRE. - Our project entitled Smart Advices For
Evaluation Systems (SAFES) was well ranked by an
independent scientific committee so that we are
currently, starting a two years feasibility study
that will allows us - To create a method of audit of standardized
testing systems - To improve the quality control procedures to be
implemented - To define a typology of subcontracted services
- To realize a market study.
- The creation of the spin-off company should
coincide with the finalisation of the e-CQCST
platform.
20VI. First results First interfaces of the
e-CQCST platform
21VI. First results First interfaces of the
e-CQCST platform
- Management of the e-CQCST users by the test
supervisor
22VI. First results First interfaces of the
e-CQCST platform
- 1st module ANALYSIS - specification table PE
are crossed with CP
23VII. Conclusions
- Docimological frameworkThe CCQCST model is
relevant and well appreciated in the context of
the ten companies and organisations which are our
partners in this project - Ergonomics aspectsThe first results show that
the ergonomics of the first interfaces facilitate
the production of specification tables related to
the first step of the standardized tests
construction - Economical valorization frameworkThe surprising
immediate docimological subcontracting services
requested by four of the ten companies (also the
acceptation of the SAFES project) shows that
there are possibilities for economical
valorisations of the results of this project. - Users frameworkThe seven categories of users
roles we defined were validated by our partners.
This clarification should contribute to highlight
the complementarities and the responsibilities of
the persons who are involved in standardized test
creation.
24Thank you for your attention ! For more
information concerning the electronic
Construction and Quality Control in Standardized
Tests project (e-CQCST) do not hesitate to
contact me jl.gilles_at_ulg.ac.be http//www.smar
t.ulg.ac.be/earli-2005/e-CQCST.ppt