The Action Language Compatibility Effect in American Sign Language

1
The Action Language Compatibility Effect in
American Sign Language

• Meylysa Tseng
• University of Hawaii at Manoa

2
Acknowledgments

• Ben Bergen
• Amy Schafer
• Victoria Anderson
• Kathryn Wheeler
• Avis Chan
• Michigan ASL Web Browser
• Sign Language friends

3
What will be discussed

• The Action-Sentence Compatibility Effect
• Sign Language
• Motor Simulation in Sign Language

4
Language affects hand motion

• Action-Sentence Compatibility Effect (ACE)
  Glenberg and Kaschak (2002)
• Find that hearing sentences which imply motion in
  a forward or a backward direction influence how
  quickly the hand moves in the same or opposite
  direction
• Supports the theory that language uses some of
  the same brain circuitry as the movement it
  represents. Motion language might evoke a
  simulation of motion.

5

Mechanics of ACE Experiment

• They used a button box with 3 buttons, one was in
  the middle, the other required movement forward,
  the other required movement backward
• Subjects heard English sentences which implied
  forward or backward movement and had to decide if
  they were sensible or not.
• Open the door. (concrete movement backward)
• Liz told you a story. (metaphorical movement
  forward)

6
Finding the ACE

• Lets say the forward button is sensible and
  the backward button is not sensible
• Subject presses and holds their finger on the
  middle ready button
• Subject hears Open the Door
• Subject lifts finger and moves forward to hit the
  sensible button
• ACE If the button marked sensible were the
  backward button, they would be slower in their
  release.

7
Results from the ACE Experiment

• When subjects saw sensible sentences which
  implied action in one direction, they had
  difficulty if making the sensibility judgment
  required moving in the opposite direction.
• The effect occurred for release time, from the
  middle button.

8
In other words

• The meaning of the sentence primes hand motion.
  If the meaning implied is motion that is
  compatible with your hand movement, there will be
  a facilitation effect on release time.
• The ACE findings seem to indicate that our
  movement is affected by language. This is
  consistent with the theory that some of the same
  cells activated to perform movement are also used
  to understand language about movement.

9
Motor control in Language

• Language can affect our motor control
• What if language itself involved movement?
• Spoken Languages - When we talk we are moving our
  tongue and various parts of our vocal tract
• Signed Languages offer an interesting twist,
  when we sign we move our hands and upper body

10
Implications of Sign Language on ACE

• If direction implied in spoken language affects
  movement, then what about in a language where
  meaning about motion coincides with actual
  movement?
• What about signs that dont imply movement in
  their semantics but phonologically consist of
  movement?

11
Research Questions

• If the ACE exists in English, does it exist in
  American Sign Language (ASL)?
• Does phonology yield ACE in ASL?

12
Sign Language Phonology

• Visual Holds and Movements (Liddell and Johnson
  1985, 1986)

ARROGANT 1st Hold
ARROGANT 2nd Hold
13
Phonology in Terms of Iconicity Auditory
modality (sound)

• Onomatopoeia Crash, Ding, Meow
• Pitch, loudness, melody
• NOTE the articulators used in speech are less
  likely to be used iconically than are the sounds
  they produce

14
Phonology in terms of Iconicity Visual modality
(body and motion)
AFFECTIONATE
15
The Implications of Phonological Iconicity
Sentences vs. Signs

• English Sentences Glenberg and Kaschak (2002)
  used sentences where the phonology did not convey
  motion, only the sentence meaning did so.
• ASL Sentences Not feasible to use in this study
  since the motions performed when signing a
  sentence cannot be kept constantTry signing a
  sentence of more than one sign with only one
  motion.
• ASL Signs Since ASL signs can be better
  controlled for consistency of movement, this
  study uses signs instead of sentences.

16
Properties of Signs

• Meaning For the purposes of this study, signs
  either convey hand/arm motion or do not convey
  motion.
• Phonology signs of interest either had forward
  movement or backward movement
• With these 2 factors we determined there to be 3
  types of signs, each of which use phonology and
  meaning in different ways.

17
Types of Signs Investigated

• Phon-Signs encode phonological motion which has
  no relation to the semantics of the sign.
• Sem-Signs encode phonological motion which
  iconically represents the concrete action meaning
  of the sign.
• Met-Signs encode phonological motion which
  iconically represents the metaphorical action
  meaning of the sign.

18
Sign Language Phon-signs

• The phonological motion has no relationship to
  the semantics of the sign.

19
Phonological Forward GIRL
20
Phonological Backward HOME
21
Sign Direction

• Fosigns Signs that have phonological hand
  motion forward
• Backsigns Signs that have phonological hand
  motion backward

22
Sign Language Sem-signs

• The phonological motion iconically represents
  hand motion encoded into the semantics of the
  sign.

23
Semantic Forward THROW
24
Semantic Backward ACQUIRE
25
Sign Language Met-signs

• The motion iconically represents metaphorical
  motion encoded into the semantics of the sign.

26
Metaphorical Forward BAWL-OUT
27
Metaphorical Backward BEFORE
28
The effects of Phonology and Semantics on Motor
Coordination

• ACE was demonstrated to occur for comprehension
  of English sentences implying movement
• Signs in American Sign Language implying movement
  have not yet been tested to see if they show ACE,
  which we must now rename Action-Language
  Compatibility Effects (as opposed to
  Action-Sentence)
• Signs in ASL which do not imply movement, but
  which phonologically consist of movement have
  also have not been tested

29
Method

• Reproduced Glenberg and Kaschak (2002) using ASL
  sign matching task this is more feasible than a
  sentence sensibility task
• Pairs of signs were shown to subjects.
• Matching condition (critical) phon/phon,
  sem/sem, met/met (22 each for 66 total)
• Non-matching condition (filler) phon/sem or met,
  sem/phon or met, met/sem or phon (22 each for 66
  total) same sign types were matched
  (Fosign/Fosign and Backsign/Backsign)

30
Trial Procedure

• Fixation Cross appears
• Subject presses and holds the h key
• Subject sees first ASL sign which is a movie
  consisting of four frames (450ms)
• Subject sees visual mask (1s)
• Subject sees second ASL sign (450ms)
• Subject lifts finger and presses the SAME
  (critical matching condition) or DIFF
  (non-critical non-matching condition) key

31
4 Experiment Versions
Block A 1/2 of (22 phon 22 sem 22 met (66
total) matching 22 phon 22 sem 22 met (66
total) nonmatching pairs) Block B the remaining
half
32
Experiment Run Procedure

• Practice session (80 required)
• 1st half of 66 trials (33 matching (critical), 33
  nonmatch (filler))
• Score given
• Experimenter changes buttons so Response
  direction is now opposite
• Second Practice Session
• 2nd half of 66 trials
• Score given

33
Hypothesis

• Subjects Reaction Times (RTs) should yield ACE
  for Met-signs and Sem-Signs.
• Subjects RTs may or may not show ACE effects for
  Phon-signs.

34
Subjects

• Ages 19 70 yrs
• Deaf or partially deaf
• Most speak some English with a hearing aid
• All can read English
• Some had knowledge of other signed languages
  (BSL, JSL, etc.)
• Live in Oahu
• Out of 46 subjects used 29
• Computer crashed
• Left handed
• ASL lt 5 years
• RTs over 3 STDEVs from the mean
• Accuracy rate lower than 80

35
Analysis

• Release RT (time to release the h key) 3-way
  Repeated Measures ANOVA for Subjects
• Response RT (time to hit the a or key)
  3-way Repeated Measures ANOVA for Subjects

36
Predictions

• Since Glenberg and Kaschak (2002) found that
  sem-signs and met-signs both showed the ACE, we
  expect to find an interaction between sign dir
  and hand motion.
• A 3 way interaction effect between Sign type x
  Sign dir x Hand motion could indicate that
  different sign types (phon, sem and met) act
  differently with respect to ACE.

37
Release Near Significant Effect of Sign Type x
Sign Direction x Hand Motion
Sign dir x Hand motion x Sign type F1 (2, 52)
1.92, p 0.16
38
Response Not so near significant Effect of Sign
Type x Sign Direction x Hand Motion
Sign dir x Hand motion x Sign type F1 (2, 52)
0.43, p 0.65
39
Discussion

• Thus, the 3-way interaction between sign type,
  sign direction and hand motion was nearly
  significant.
• By lumping sem-signs and met signs together we
  can now see if, as a group, they yield the ACE
  (Sign dir x Hand motion).
• We can also see if phon-signs yield the ACE.

40
Release Near Significant interaction Phon vs
Sem/Met

• signdir x motion x signtype F1(1, 26) 2.36, p
  0.14

41
Response ACE, phon vs. sem/met-signs
Sign dir x Hand motion F1(1, 26)4.12, p .05
42
Response ACE, Met-Signs and Sem-Signs
Sign dir x Hand motion F1(1, 26) 6.83, p
0.02
43
Response ACE Near Significant for Sem-Signs,
Sign dir x Hand motion
Sign dir x Hand motion F1 (1, 26) 1.89, p 0.18
44
Response No ACE for Phon-signs, sign dir x hand
motion
Sign dir x Hand motion F1 (1, 26) 0.53, p 0.48
45
Discussion

• Response vs. Release Significant interaction
  effects were only found for the response.
• Not enough subjects (Release ACE F value was
  over 2)
• Sentence sensibility rating task requires
  thinking about semantics before release
• Sign matching task involves a more automatic
  response, where meaning effects would possibly
  show up after the decision is made, in the
  response RT
• Why no significant ACE effects for Sem-signs?
• Probably not enough subjects (F was almost 2)
• Perhaps subjects didnt recognize some of the
  Sem-signs (Hawaiian ASL dialect vs. Michigan
  State ASL Browser)

46
Phon-Signs different from Met-Signs and Sem-Signs

• The experiment shows ACE only occurs for
  semantically meaningful phonological motion

47
Backsigns Have Longer Release RTs
Sign dir F1 (1, 26) 18.00, p 0
48
Why longer release RTs for Backsigns?

• Fosigns have place and handshape encoded at the
  start as opposed to some Backsigns which encode
  this information at the end of the sign,
  therefore perhaps it took signers longer to
  recognize Backsigns.
• Some of the Backsigns had shorter distances
  covered than Fosigns, therefore the movement
  between holds was more minute and harder to see,
  possibly making it more difficult for signers to
  recognize the sign.

49
Summary of Results

• Three way interaction effect of Sign type x Sign
  dir x Hand motion was nearly significant for
  Release and Response RTs
• ACE was almost significant in Release RTs for
  sem/met (in sem/met vs. phon)
• Two-way interaction effect for Sign dir x Hand
  motion (ACE) for Response RTs was significant for
  sem/met (in sem/met vs. phon) and met
• Sem-signs had Response RTs which show nearly
  significant ACE
• Phon-signs had Response RTs which showed no ACE
• Backsigns cause a significantly longer release RT

50
Conclusions

• Evidence for an Action-Language Compatibility
  Effect (as opposed to Action-Sentence)
• ACE only occur when motor actions represent
  meaning
• Motor action which is phonological and not
  semantic perhaps activates areas of the brain
  different from motor action which has semantic
  significance

51
Future Research

• Run the experiment on more subjects (deaf,
  partially hearing and hearing)
• Conduct a norming study to determine that Phon,
  Met and Sem signs are all valid
• Conduct a norming study to control for dialectal
  variation in signing
• Test other types of signs (deictics, phonetic vs.
  phonological motion)

52
Thank you
53
References

• Glenberg, A. M. and M. P. Kaschak. (2002).
  Grounding language in action. Psychonomic
  Bulletin Review.
• Liddell, Scott and Robert Johnson. (1985).
  American Sign Language The phonological base.
  Unpublished ms.
• Liddell, Scott and Robert Johnson. (1986).
  American Sign Language Compound Formation
  Processes, Lexicalization, and Lexical
  phonological remnants. Natural Language and
  Linguistic Theory 4. 445-513.

54
Conditions

• 66 signs total
• 11 Phon-Fosigns, 11 Phon-Backsigns
• 11 Met-Fosigns, 11 Met-Backsigns
• 11 Sem-Fosigns, 11 Sem-Backsigns
• Matching
• Critical trials
• Either responded by moving FORWARD or BACKWARD
  the body
• Nonmatching
• Noncritical trials
• Fosigns matching with Backsigns and vice versa
• Different types matched together randomly
• i.e. Sem-Fosign with Phon-Fosign

55
Other Results

• Release 1-way ANOVA by Items F(2, 57) motion x
  signtype x signdir 1.42, p .25
• Response 2-way ANOVA (met) by Subjects ACE
  signdir x motion F(1, 26) 9.29, p lt 0.01
• Release 3-way ANOVA (phon, sem/met) Response
  2-way ANOVA (sem) signdir x motion F(1, 26)
  1.87, p0.18
• Response 2-way ANOVA (semet) signdir x motion
  F(1, 26) 6.83, p 0.02
• Release 2-way ANOVA (phon) signdir x motion F(1,
  26) 1.67, p 0.21
• Release 2-way ANOVA (semet) signdir x motion F(1,
  26) 1.87, p 0.18
• Release 2-way ANOVA (sem) signdir x motion F(1,
  26) 2.07, p 0.16
• Release 3-way ANOVA by Items (p,s,m) Sign dir x
  Hand motion F2(1, 57) 3.78, p 0.06

56
Release Near Significant ACE for Sem/Met Signs

• signdir x motion F1(1, 26) 1.87, p0.18

57
Method - Implemenation

• E-Prime on laptop
• Detachable keyboard placed perpendicular to the
  body, with the a key closest to you. The h
  key is the middle key and the key is the key
  farthest from you.
• Locations office cubicles and secluded areas
  outside

58
Results

• AFFECTIONATE, BINARY and GOODLUCK
• Eliminated because RTs were beyond 3 STDEV from
  the mean for item RT means
• RTs beyond 3 STDEV replaced with the Maximum
  value (mean3 x STDEV)