Soft Skills Systems / Teaching / Assessment

1
Soft SkillsSystems / Teaching / Assessment

• Kostas Kechagias

Presentation in 2nd MASS meeting Work Package 4,
Needs Analysis / Piloting
June 2010
2
Contents

• Skills and Skills Frameworks
• Teaching Skills and Competences
• Skills Assessment
• Experimental Methodology
• Conclusions/Homework

3
Skills and Skills Frameworks (1/13)

• towards definitions (DeSeCo)
• great interest for identification of common
  skills
• must be connected to success in life
• the definition should address how they are used
• Definition skills or abilities individuals need
  in order to achieve success in life, within the
  context of their socio-cultural milieu, through
  adaptation to, shaping of, and selection of
  environments
• Often they are used interchangeably or in a vague
  sense

4
Skills and Skills Frameworks (2/13)

• a proliferation of efforts to develop lists of
  skills, knowledge, and competencies necessary for
  success in the workplace and society
• basic skills are not sufficient for success in
  the workplace
• employability skills
• enabling skills, generic skills, core
  skills, key competencies, essential skills,
  and necessary skills.
• methodology start from a broad definition and
  then survey or observes workers, supervisors, and
  experts to determine what skills are common

5
Skills and Skills Frameworks (3/13)

• SCANS
• 1991, US president and Education Secretary,
  America 2000
• seven skill groups
• Influence Organizational Effectiveness and
  leadership
• Group effectiveness Interpersonal skills,
  negotiation and teamwork
• Personal management Self-esteem, goal
  setting/motivation, and personal and career
  development
• Adaptability Creative thinking and problem
  solving
• Communication Listening and oral communication
• Competence Reading, writing and computation
• The foundation Learning to learn.

6
Skills and Skills Frameworks (4/13)

• Michigan
• Center for Career and Technical Education,
  Michigan State University
• human relations skills
• Positive work attitude,
• Teamwork
• Listening
• Honesty and Idengrity
• Demonstrate initiative
• Handling pressure tension
• Communication
• conceptual skills
• technical skills

7
Skills and Skills Frameworks (5/13)

• Scottish Qualifications Authority, 2008
• Problem Solving
• Communication
• Numeracy
• Information and Communication Technology
• Working with Others
• Component Working Co-operatively with Others
• identifying, with support, your own role and the
  roles of other people, and helping to achieve a
  shared goal
• to
• analyzing the roles and behavior of others and
  adapting your own behavior to deal with the
  complexity of changing and challenging dynamics

8
Skills and Skills Frameworks (6/13)

• DeSeCo (Definition and Selection of Competencies)
• OECD, 1997
• A competency is more than just knowledge and
  skills. It involves the ability to meet complex
  demands, by drawing on and mobilising
  psychosocial resources (including skills and
  attitudes) in a particular context.

9
Skills and Skills Frameworks (7/13)

• Using Tools Interactively
• use language, symbols and text interactively
• use knowledge and information interactively
• use technology interactively
• Interacting in Heterogeneous Groups
• relate well to others
• cooperate
• manage and resolve conflicts
• Acting Autonomously
• act within the big picture
• form and conduct life plans and personal projects
• assert rights, interests, limits and needs

10
Skills and Skills Frameworks (8/13)

• ALL (Adult Literacy and Life skills survey
  project)
• OECD, 2005
• employability skills derived from job and task
  analysis and psychological theories of
  intelligence
• scientifically convincing theory and an
  established literature
• accompanying tradition of measurement
• could be learnt and, therefore, taught
• had a direct impact on the social, health,
  educational or economic life chances of
  individuals

11
Skills and Skills Frameworks (9/13)

• European Qualification Framework
• Education and Training 2010 (2004)
• communication in the mother tongue
• communication in foreign languages
• mathematical competence and basic competences in
  science and technology
• digital competence
• learning to learn
• social and civic competences
• sense of initiative and entrepreneurship
• cultural awareness and expression

12
Skills and Skills Frameworks (10/13)

• MASS Framework
• we dont have much information

Manners
Ownership of tasks
Attendance
Motivation
Professionalism
Work output
Conduct in workplace
Timekeeping
Verbal Communication
Organisation/ planning
Team-working/ Respect
Helping others
Conscientiousness
Ability to ask for help
Adaptability/ Flexibility

• dont seem to fit in any studied framework

13
Skills and Skills Frameworks (11/13)

• No grouping of skills is provided
• Some skills may be fit in more than one
  categories in other skill frameworks, e.g.
• Motivation
• Taking on new challenges, working hard to achieve
  goals, thinking of new ways to do things
• Under SCANS contains elements from categories
• Personal Qualities (taking on new challenges)
• Thinking Skills (thinking of new ways to do
  things)
• Seems to be closely related to Proffesionalism
• Working to a high standard, being consistent in
  attitude (not allowing emotions or personalities
  to influence you)

14
Skills and Skills Frameworks (12/13)

• Most frameworks are concerning higher education
  graduates
• One may argue that the target group is different,
  so the skills to be considered are different
• however, this should be established in the
  grounds of empirical research of literature
• The skills should be defined in such a way as to
  minimize overlap
• Failure to do so may have impact to teaching, but
  mainly to assessment.

15
Skills and Skills Frameworks (13/13)

• Questions
• What is the framework on which the MASS skills
  are based?
• What justifies the selection of the particular
  skills as the most important ones?
• If there is not a given one, referred from the
  literature, what are the empirical studies that
  support the selection?
• Are the skills chosen in such a way as the
  interdependencies are minimized?
• Is the selection justified from the view point of
  the assessment?

16
Teaching Skills and Competences (1/9)

• Gonczi, 2003 (DeSeCo)
• The best examples active independent learning,
  especially in ways that simulated contexts they
  might experience in later life
• those that integrated the key competencies into
  the teaching of other material, rather than
  attempting to teach them as stand-alone entities
• seemed to be more success in student outcomes
  when the key competencies were made explicit
• in many instances, it was not possible to
  separate out the key competencies in the process
  of teaching and learning.
• very few of the projects in vocational education
  specifically assessed the acquisition of key
  competencies directly. The most comprehensive
  attempt to assess them was via a portfolio of
  evidence which students collected for themselves.
  

17
Teaching Skills and Competences (2/9)

• Welsh Canning, 2003 (SQA)
• The prospect of the isolation of Core Skills, as
  a separately taught and assessed subject, would
  appear to be unpopular with candidates. From the
  literature, it can be seen that this is also
  unpopular with employers.
• Completely separate delivery and assessment often
  not popular with candidates as its tantamount
  to a return to school

18
Teaching Skills and Competences (3/9)

• Shuman, et. all., 2005 (ABET)
• The choice of team task depends on the
  educational objectives
• if the objective is to have students experience a
  specific aspect of teaming such as brainstorming,
  specific group exercises new ideas can be used
• if the objective is for students to experience a
  broad range of team processes and behaviors, then
  more complex activities are suitable

19
Teaching Skills and Competences (4/9)

• Shuman, et. all., 2005 (ABET)
• Two guiding principles should be followed in
  choosing activities fidelity and complexity
• Fidelity similarity of the training situation to
  the students present and future working
  conditions.
• The higher the fidelity, the more superior the
  transfer of learning to the workplace
• Complexity is defined by two subfactors task
  interdependence and cognitive effort. The more
  complex the activity, the more team skills are
  required by the participant.

20
Teaching Skills and Competences (5/9)

• Shuman, et. all., 2005 (ABET)
• advantages and disadvantages
• fidelity/high complexity activities
• most resemble real workplace conditions,
• typically are more difficult for the instructor
  to manage,
• resource intensive,
• time consuming for the student.
• lower in fidelity/ low complexity activities
• typically more structured and easier to
  administer,
• may be perceived as less relevant by the student,
  
• the experience having less of a learning impact.
  Finally,
• may not challenge the team nor provide the
  environment necessary for intense interaction
  among team members

21
Teaching Skills and Competences (6/9)

• Teaching methods (Szul, 2002)
• Coaching
• Project Work (e.g. Newsletter)
• Portfolio
• Job-shadowing experiences
• Videotapes/Observations
• Interviews
• Case studies
• Role playing

22
Teaching Skills and Competences (7/9)

• MASS Teaching
• seems to be standalone course
• justification needed
• Interdisciplinary
• 18 weeks of approximately 4 hours/week
• partners should be consulted for the feasibility

23
Teaching Skills and Competences (8/9)

• The activities that the 1st learning byte
  contains belong to the following categories
• Presentation 4 times
• Group discussion 2 times
• Directed Discussion (Class Discussion) 6 times
• Quiz 2 times
• Observation 1 time
• Individual work 1 time
• Personal discussion with the teacher 1 time
• Mainly teacher centered designed
• Hopefully, your grades will be improved

24
Teaching Skills and Competences (9/9)

• the design of the lessons seems a lot like the
  designs of the vocational training
• Question about their efficiency
• Excellent quality of materials, within the
  concept
• instruction based not project based
• question whether we train on the knowledge of
  soft skills rather than soft skills themselves
• In some places, as in formation of student
  groups, more elaboration is needed

25
Assessment (1/14)

• indicators and outcomes
• indicators are the means by which we can measure
  whether the outcomes have been achieved
• achievements which may indicate acquisition or
  progress towards an outcome
• Not all indicators will be suitable for all
  target groups, and some will be target group
  specific.

26
Assessment (2/14)

• Methods for collection of information on soft
  outcomes
• Individual action planning, personal action
  planning and goal setting
• Reviews between trainers/assessors and clients to
  record soft outcomes
• Daily diary or personal journal
• In-depth reflection during or after the course
• Recorded observations of group or individual
  activities
• Presentation of material in a portfolio
• Tests

27
Assessment (3/14)

• requirements from the tools
• reliability
• Validity
• Requirements from the definitions
• two ends
• so general that it becomes totally abstract and
  refers more to a system of values than to
  abilities
• less comprehensive in ones ambitions and is
  content with assessing a few specific
  competencies in a necessarily limited context the
  

28
Assessment (4/14)

• Competencies do not exist in themselves they can
  only be demonstrated when an individual is acting
  in a specific context.
• pieces of research have shown that the subjects
  performance is largely dependent on the contents
  of the task, on any previous training the subject
  may have had, on the type of strategy resorted
  to, and on the formal characteristics of the task
• the same tasks undertaken in different
  situationssuch as work situations or job
  interview situationscan yield very different
  results

29
Assessment (5/14)

• Teamwork (Baker, et al., 2005) (ALL Framework)
• Focus towards identifying the specific competency
  requirements of team members
• competencies a cluster of related knowledge,
  skills, and attitudes that affects a major part
  of ones job
• is correlated with performance on the job
• can be measured against well-accepted standards
• can be improved through training and development.

30
Assessment (6/14)

• three types of competencies that are central for
  effective teamwork Cannon-Bowers et al. (1995)
• team knowledge competencies,
• the principles and concepts that underlie a
  teams effective task performance
• team skill competencies,
• learned capacity to interact with other team
  members at some minimal proficiency level
• adaptability, situation awareness, performance
  monitoring/feedback, leadership, interpersonal
  relations, coordination, communication, and
  decision making
• team attitude competencies
• an internal state that influences a team members
  choices or decisions to act in a particular way
• Positive attitudes toward teamwork and mutual
  trust among team members

31
Assessment (7/14)

• ALL questionnaire for teamwork
• Evaluation of teamwork requires observation
• Target group International, adults
• Examines
• knowledge about team skills,
• attitudes toward teamwork
• historical experience with teamwork

32
Assessment (8/14)
33
Assessment (9/14)

• Zhuang, et. all., Roberts, 2008 (ETS)
• Questionnaire
• target at high school students (mean age 16.10)
  undertaking Ford PASS courses
• consists of reliable and qualitatively distinct
  factors
• involves a variety of methods
• has demonstrable validity evidence

34
Assessment (10/14)

• Contains
• Self-report ratings (57 items)
• cooperation with others (15 items),
• advocate and influence (12 items),
• resolve
• conflict/negotiate (17 items),
• guiding others (13 items).
• Situational judgment tests (SJTs)
• eight scenarios describing various situations and
  required participants to evaluate the
  effectiveness of four reactions to each situation
• Teacher ratings

35
Assessment (11/14)

• MASS assessment
• one might form the impression of an underlying
  assumption that each skill, can be assessed
  separately
• Is it possible? Interrelated skills?
• Some skills, as motivation is difficult to
  assess outside a particular framework
• Need for clarification of definitions?
• Development of indicators?
• Framework?

36
Assessment (12/14)

• the main assessment tool proposed is as self
  reference questionnaire
• students mark their grades against each skill,
  every week
• They can see the grades they put to themselves in
  the previous weeks
• aim being that your scores will have improved -
  biased toward to a false improvement of grades?
• What about the students that they do not (or they
  thing they do not) improve? motivation for more
  or drop out?

37
Assessment (13/14)

• three entities involved in the course, the
  students, the teacher(s) and the materials
• Student self evaluation
• Student evaluation from the students (peer
  evaluation)
• Student evaluation from the teacher(s)
• Teacher self evaluation
• Teacher evaluation from the students
• Material and course quality evaluation from the
  students
• Material and course quality evaluation from the
  teachers.

38
Assessment (14/14)

• questionnaires are convenient, but others, like
  portfolio or in depth interviews may be
  considered
• the partners should decide about the aims and the
  methodology of the evaluation
• If we decide to confine ourselves in a given
  framework, the uses of readymade and tested tools
  are also possible
• other aspects of influence
• self-esteem and self-image

39
Experimental Methodology (1/1)

• The proposed methodology is the classical
  experimental one
• Two groups, the experimental and the control one
• Initial Assessment
• Intervention
• Final Assessment
• What about long term results?

40
Conclusions/Homework (1/3)

• Does MASS framework needs clarification and
  justification?
• Is the proposed schedule 18 weeks X 4 hours
  suitable for the partners?
• Each partner should report on this
• Suggest modifications
• Two alternatives
• All the same schedule Comparable results
• Different schedules Test more than one
  variations of program to see which works best.

41
Conclusions/Homework (2/3)

• Teaching organization
• How many students
• How many classes
• What age
• What educational background
• Suggested period of teaching
• what else?
• Decision about assessment
• Target of assessment
• Ages of students
• Assessment procedure questionnaires only or
  something more?
• Each partner studies and comments proposed
  questionnaires and maybe contributes with more
• what else?

42
Conclusions/Homework (3/3)

• Deadline?
• What about end of June?

43

• Thanks for your patience!!!