Helper Joints: Advanced Deformations on RunTime Characters

1
Helper JointsAdvanced Deformations on RunTime
Characters

• Jason Parks
• Character Technical Director
• Sony Computer Entertainment America

2
Contacts, Reference, Credits

• Email
• jason_at_jason-parks.com
• jason_parks_at_playstation.sony.com
• Webpage (Helper Joints-GDC 2005)
• http//www.jason-parks.com/HelperJoints
• PAN Arts AVIs by Warwick Mellow
• Human Model by Sven Jenson via Miller/Thuriots
  MasterClass 2003 (Character Creation Toolkit)
• BodyBlend render by Zach Gray
• Wrinkle RD by Tyler Crook (SCEA)

3
Intended Audience

• Advanced Requires experience and familiarity
  with subject.
• Maya Technical Artists who Bind Characters for
  RunTime or Offline usage
• Paint weights (Artisan and Component Editor)
• Apply deformers
• Use Set Driven Keys
• Write Expressions
• Use Utility Nodes

4
Topics

• What are Helper Joints?
• Definition
• Why use Helper Joints?
• New uses for Extra Joints
• Smooth binding sucks Where to use Helper Joints
• How to use Helper Joints?
• Our Goals, Disclaimer, Fusiforms
• Research The Basic Building Blocks
• RunTime Rigs
• SetDrivenKeys
• Expressions
• Constraints
• Non-RunTime Rigs
• Baking Out
• Spline IK
• Jiggle Deformer
• Automating Production
• Scripting
• Workflow

5
Stop Talking, No more text, show some Moving
Pictures!
6
Helper Joints Defined

• Helper Joints are secondary joints offset from
  the standard hierarchy which will translate,
  rotate, or scale to achieve a hand sculpted
  affect on an area of vertices.
• They can be within the primary hierarchy
  (parented, grouped, rigged), or outside of the
  primary hierarchy (point or parentConstrained)
• They can be controlled by SetDrivenKeys,
  Expressions, Simple Rigs (constraint system), or
  Super Advanced Rigs (really any simulation you
  can think of) that are baked-out.

7
Why use any Extra Joints?
Unlike characters rigged, bound, and rendered in
Maya or any other high-end 3D package, your
characters at run-time need to follow a very
specific rule set depending on your Game Engine.
Because all run-time engines already support
joints, it is a tool we can exploit and push to
its limits.

• Traditional usage of Extra Joints to animate
  Props and Accessories
• Extra Helper Joints to fix Smooth binding
• Extra Helper Joints to create the next level in
  realism

8
Traditional Extra Joints

• Traditional usage of Extra Joints for Props and
  Accessories
• Props, Rope
• Hair, Ponytails
• Capes, Necklaces, Clothing
• Fat

9
New Ideas for Extra Joints

• Extra Helper Joints to create the next level in
  realism
• Skin sliding
• Wrinkles
• Fat/Muscle/Flesh Jiggle
• Muscle bulging

10
Fix Smooth binding

• Extra Helper Joints to make skin transform
  properly
• Volume maintenance
• Crease fixing
• Complex Areas
• Multiple joints affecting a single vert

11
Smooth Binding Limited

• Vertex deforms by a weighted combination of
  Linear Translations around a joint rotating

12
Smooth Binding Shoulder
13
Smooth Binding Problem Areas

• Elbow and Knee
• Interpenetration on the inside of the crease
• Loss of volume on the outside of the crease (i.e.
  the elbow tip or knee cap do not become
  pointy when the elbow or knee bends)
• Wrist/Forearm
• Loss of volume when the forearm/wrist is twisted
  too much down the length of the arm. (i.e. -
  Candy-wrapper affect)
• Upper Thigh/Gluteus
• Interpenetration at the bottom of the butt cheek
  when leg goes too far back
• Shoulders
• Interpenetration in armpit
• Interpenetration at the neck/shoulder junction
  area (trapezius)
• Lack of representation of scapula movements

14
Our Goals The Ideal

• What are we really trying to get?
• Reference with Motion
• Books
• Anatomy of Movement by Blandine C. Germain
• Internet
• PAN Arts by Warwick Mellow
• Software
• Absolute Character Tools for 3DSMAX

15
Our Goals

• PAN Research
• Pectorals
• Shoulders

16
Our Goals

• PAN Research
• Clavicle
• Forearm

17
Our Goals

• Muscles Independent Research

18
Our Goals

• Muscles - Absolute Character Tools

19
Disclaimer RunTime Rig Support

• Individual Developer code will dictate what will
  be evaluated in Real-Time and what type of
  relation you can specify between your primary
  skeletons animation and your Helper Joints
  transformation.
• Hopefully you can get 1 or more of these RunTime
  Rigging methods supported
• SetDrivenKeys
• Expressions
• Constraints

20
Disclaimer RunTime Rig Support

• NO
• Source Code provided for engine side.
• YES
• Techniques for art-side production of rigs (in
  Maya)
• Applicable to all games on all consoles

21
How To Get There

• Muscles are the drivers in reality so lets create
  Helper Joints that mimic the way muscles work.
• Simple FusiForm muscles are easy to mimic in Maya
  so lets create those and bind them to our skin

22
Research FusiForm Muscle

• FusiForm muscle Quick Definition
• Tapering at each end spindle-shaped.

23
Research The Building Blocks

• Lets pick the Biceps as our test case to create
  a muscle and mimic its action with a Helper Joint
  being driven by SetDrivenKeys
• Rotating
• Scaling
• Translating
• The research explores the underlying mechanisms
  of the rigs
• The failed attempts still illustrate how rigs
  can be put together

24
Research The Building Blocks

• UNDERSTANDING HOW THE RIGS WORK
• IS MORE VALUABLE
• THEN MY SUGGESTED SOLUTIONS

25
Research Biceps (NURBs)

• Muscle as Influence Object - NURBs

26
Research Biceps (Polys)

• Muscle as Influence Object - Polygons

27
Research Biceps (Short)

• SetDrivenKey w/ Single Driver Axis
• Rotating
• Short

28
Research Biceps (Long)

• SetDrivenKey w/ Single Driver Axis
• Rotating
• Long

29
Research Biceps (Scale)

• SetDrivenKey w/ Single Driver Axis
• Scaling

30
Research Biceps (Translate)

• SetDrivenKey w/ Single Driver Axis
• Translate

31
Research Upper Arm (Translate)

• Single Driver Axis
• Translate
• Works pretty good for Single driving axis!
• Warning Not anatomically correct!

32
Research Biceps (RollBone)

• SetDrivenKey w/ Single Driver Axis
• Translate
• Add Roll
• BREAKS!

33
Research Biceps (RollBone)

• SetDrivenKey w/ Two Driver Axis
• Translate
• Add Roll
• w/ Extra Keys compensating for roll
• Only fixes in when arm is straight out
• When arm is bent it still BREAKS!

34
Research Biceps (RollBone)

• Add parent joint w/ another set of driven keys
  for roll
• Rotation of Helpers parent joint is percentage
  of primary hierarchies rotation

35
Research Shoulder (Multiple Drivers)

• SetDrivenKey w/ Single Driver Axis from Two bones
• Translate
• Combination is difficult

36
Research Shoulders (Multiple Drivers)

• SetDrivenKey w/ Two Driver Axis from Two bones
• Translate
• 2 Driving bones BREAKS!(kinda)
• 2 Driving axis BREAKS!BAD!

37
RunTime Rigs SetDrivenKeys

• Summation
• SetDrivenKeys are fast, easy to use and work
  great for simple Helper Joints
• 1 driveR joint and 1 driveR axis per driveN
  Helper Joint
• Nest hierarchies of driveN Helper Joints if you
  have more than 1 driveR joint or axis

38
RunTime Rigs SDKs for Volume Preservation

• Forearm Helper Joint w/ SDKs for Scale and
  rotation

39
RunTime Rigs SDKs for Wrinkles

• Scaling of Helper Joints (RD by Tyler Crook _at_
  SCEA)

40
RunTime Rigs SDKs for Wrinkles

• Joints are scaling on 2-axis (bulging), no
  scaling on length-wise axis
• No translation or rotation

41
RunTime Rigs SDKs for Wrinkles

• Weighting is very tedious

42
RunTime Rigs SDKs for Wrinkles

• For shirt it takes many more joints because of 3
  D.O.F. and larger surface.
• 6 joints to do it right

43
RunTime Rigs SDKs for Wrinkles

• Different joints scale for each direction in each
  D.O.F.
• Weighting is time consuming

44
RunTime Rigs SDKs for Wrinkles

• Incorporation
• Could theoretically add a scaling joint as a
  child of the helper joints
• Then just weight individual vertices to these
  scaling joints to cause wrinkling
• Or just make separate scaling wrinkle joints
  wherever needed

45
RunTime Rigs

• Done with SDKs
• Start Expressions

46
RunTime Rigs Expressions (SimpleElbow)

• Simple Helper Joints which only need a
  Translation Expression
• Based on 1 axis from driving bone
• dj_helper.ty (base.ty (maxOffset (elbow.rz
  /maxAngle)))

47
RunTime Rigs - Expressions (ComplexBiceps)

• Complex Helper Joints which need Translation
  Rotation Expression
• Translation Use another expression to translate
  dj_bicep joint (parented to Bicep_Helper) for
  bulge based on elbow rotation.
• Expression 2 Describes bulge from rotation of
  elbows Z-axis
• dj_bicep.ty (base.ty (maxOffset (elbow.rz
  /maxAngle)))
• base.ty 5, maxOffset 1.8, maxAngle 110
• Rotation Joint needs to be parented to both
  shoulder and bicep-roll joints. Use Expression to
  take percentage.
• Expression 1 Describes rotation of parent joint
  around bicep-roll joint. (60 from shoulder, 40
  from bicep-roll)
• Bicep_Helper.rx ((.4 bicep.rx) (.6
  shoulder.rx))

48
RunTime Rigs - Expressions (ComplexBiceps)

• Rotation Bicep_Helper.rx ((.4 bicep.rx)
  (.6 shoulder.rx))
• Translation dj_bicep.ty (base.ty (maxOffset
  (elbow.rz /maxAngle)))
• base.ty 5, maxOffset 1.8, maxAngle 110

49
RunTime Rigs Expressions Constraints
(ComplexBiceps)

• Complex Helper Joints which need Scale Rotation
  Expressions but looking at Distance
• Requires 6 extra parent helpers (can be joints or
  nulls)
• Component 1 Scale (Bulge)
• Handled by expression based on distance of
  shock-absorber
• bicepHelper.sz (scaleFactor(1/(boneLength)))
  boneLength sqrt ((foreArmOffset.tx
  bicepOffset.tx)² (foreArmOffset.ty
  bicepOffset.ty)² (foreArmOffset.tz
  bicepOffset.tz)²)
• Component 2 Rotate
• Handled by expression base on of angle of
  bicep_roll
• Bicep_Helper.rx ((.4 bicep.rx) (.6
  shoulder.rx))
• Requires 2 pointConstraints

50
RunTime Rigs Expressions Constraints
(ComplexBiceps)

• Scale Expression based on distance

51
RunTime Rigs - Expressions

• Summation
• Very difficult to describe a motion of a Complex
  Helper Joint based on 2 or more axis from driving
  joint. Even MORE difficult when there are
  multiple bones acting as drivers.

52
RunTime Rigs

• Done with Expressions
• Start with Constraints

53
RunTime Rigs - Constraints

• pointConstraint
• Good for position only
• orientConstraint
• Good for orientation only
• parentConstraint
• Good for both position and orientation
• aimConstraint
• Robust and elegant
• vertexConstraint
• Allows the use of muscles

54
RunTime Rigs - Constraints

• AIMCONSTRAINT
• IS
• DA BOMB!

55
RunTime Rigs Constraints (ComplexBiceps)

• aimConstraint pointing at forearm
• Arc problem
• Handles rotation of bicep roll on extension but
  not flexion

56
RunTime Rigs Constraints (ComplexBiceps)

• aimConstraint flexion problem
• Handles rotation of bicep roll on extension but
  not flexion

57
RunTime Rigs Constraints (ComplexBiceps)

• aimConstraint flexion problem handled on XZ plane
  w/ expression for Bulge
• Bicep_HelpersParent.scale (baseScale
  (maxOffset (elbow.rz /maxAngle)))
• i.e. - baseScale 1, maxOffset .3, maxAngle
  110

58
RunTime Rigs Constraints (ComplexBiceps)

• aimConstraint still broken on Y plane during
  flexion

59
RunTime Rigs Constraints (ComplexPectoralis)

• Requires 2 extra parent helpers (can be joints or
  nulls)
• Requires 1 aimConstraint

60
RunTime Rigs Constraints (ComplexArmPit)

• Requires 2 extra parent helpers (can be joints or
  nulls)
• Requires 1 aimConstraint

61
RunTime Rigs Constraints (ComplexPectoralis
Armpit)

• Requires 3 extra parent helpers (can be joints or
  nulls)
• Requires 2 aimConstraints

62
RunTime Rigs Constraints (SimpleNeck)

• Look at normal smooth bind

63
RunTime Rigs Constraints (SimpleNeck)

• NURBs muscle as Influence Object

64
RunTime Rigs Constraints (SimpleNeck)

• Requires 1 helper joint and 1 target null for
  aiming
• Requires 1 aimConstraint

65
RunTime Rigs Constraints (SimpleNeck)

• Comparison

66
RunTime Rigs Constraints (SimpleNeck - Back)

• Requires 1 helper joint and 1 target null for
  aiming
• Requires 1 aimConstraint

67
RunTime Rigs Constraints (SimpleNeck - All)

• Requires 4 helper joints w/ 4 extra target nulls
• Requires 4 aimConstraints

68
RunTime Rigs Constraints (SuperComplex
Scapula)

• Look at normal smooth bind

69
RunTime Rigs Constraints (SuperComplex
Scapula)

• What is scapula really doing?

70
RunTime Rigs Constraints (SuperComplex
Scapula)

• Comparison

With Rig
Normal Binding
71
RunTime Rigs Constraints (SuperComplex
Scapula)

• Shrugging

72
RunTime Rigs Constraints (Exp./SDK)
(SuperComplex Scapula)

• Requires 2 helper joints and 2 target nulls for
  aiming
• Requires 2 aimConstraints 1 pointConstraint
• Requires expression w/ if statement
• If (lShoulder.rotateY gt0)
• l_scapulaConst.rotateX ( scaleFactor
  (lShoulder.rotateY/maxAngle))
• Else
• l_scapulaConst.rotateX 0
• Ex scaleFactor 30 degrees and maxAngle 90
  degrees.
• OR SetDrivenKey

73
RunTime Rigs Constraints (Exp./SDK)
(SuperComplex Scapula)

• Explanation of (SDK or If statement in
  expression)

74
RunTime Rigs Constraints (Exp./SDK)
(SuperComplex Scapula)

• Pros Full Range Looks Great
• Cons Complex and Weighting is Hard!

75
RunTime Rigs Expression/SDK (Complex
ArmPitBack)

• Normal armpit crease

76
RunTime Rigs Expression/SDK (Complex
ArmPitBack)

• New Helper Joint added

77
RunTime Rigs Expression/SDK (Complex
ArmPitBack)

• Translation of Helper Joint tip driven by
  expression or SetDrivenKey from shoulder rotation
  
• Requires 1 extra parent Helper Joint w/
  percentage rotation of shoulder driven by Rotate
  expression or SetDrivenKey

78
RunTime Rigs Expression/SDK (Complex
ArmPitBack)

• armPitParent Joint rotating 60 of shoulder joint

79
RunTime Rigs Exp./SDK (Complex ArmPitBack)

• armPitTip joint translating down its local X Y
  as a function of shoulder rotation

80
RunTime Rigs Exp./SDK (Complex ArmPitBack)

• Requires 1 simple expression for parent Joints
  rotation
• HelperParent.rotateY (offset angle (( arm
  rotate) lShoulder.rotateY)))
• Ex - dj_armPitBackParent.rotateY (-35 (.6
  lShoulder.rotateY))
• Requires 1 expression for translation of tip w/
  if statement
• If (lShoulder.rotateY lt0)
• HelperTip.translateX (base.tX
  (scaleFactorX(lShoulder.rY/maxAngle)))
• HelperTip.translateY (base.tY
  (scaleFactorY(lShoulder.rY/maxAngle)))
• Else
• HelperTip.translateX baseX
• HelperTip.translateY baseY
• Ex scaleFactorX 2 units, scaleFactorY -4
  units and maxAngle 65 degrees.

81
RunTime Rigs Exp./SDK (Complex ArmPitBack)

• Weighting is powerful and tricky. Some vertices
  now have 5 different weights.

82
RunTime Rigs Scaling Entire Rig

• What if character needs to be scaled? (i.e.-
  player defined proportions)
• A simple Group node above the rig mechanism in
  Mayas hierarchy will scale easily
• The Trick is figuring out how much to scale. This
  is dependent on what space the helper joint
  deforms, and how that space has been scaled.

83
RunTime Rigs

• Done with Constraints
• Start with Muscles

84
RunTime Rigs - Muscles

• Muscles at RunTime allows you to constrain Helper
  Joints to the muscle surface and get the effect
  of the muscle
• Requires some sort of geometryConstraint

85
RunTime Rigs Constraints Bind Blendshape
(MusclesBiceps)

• Deformer Object is low-count poly muscle rigid
  (or smooth) bound to two joints to handle
  twisting of bicep
• Requires a blendShape on deformer object to
  handle bulge.
• Note blendShape is not required on main
  skinning surface.
• Complex Helper Joint is VertexConstrained to
  vertex on Deformer Object

86
RunTime Rigs Constraints Bind Blendshape
(MusclesBiceps)

• VertexConstrained

87
RunTime Rigs Constraints Bind SDK
(MusclesBiceps)

• Poly Muscle (Simplest Form)
• 1 triangle Rigid (or Smooth) Bound
• Handles rotation of bicep
• SDK (or simple expression 1 driving bone/axis, 1
  driven translating axis) driving single vertex
  for bulge
• Complex Helper joint VertexConstrained to
  single vert
• Finally skinning layer

88
RunTime Rigs Constraints Bind SDK
(MusclesBiceps)

• 1 Poly (Simplest Form)
• VertexConstraint

89
RunTime Rigs Constraints Bind
(MusclesPectoralis)

• Deformer Object is low-count poly muscle rigid
  (or smooth) bound to three joints to handle
  complex Pectorals motion
• Requires MuscleDriver parent joints to be offset
  from main skeletal joints
• MuscleDriver child joints are pointConstrained
  to parent joints to avoid rotation of parent
  joints
• Complex Helper Joint is VertexConstrained to
  vert on Deformer Object

90
RunTime Rigs Constraints Bind
(MusclesPectoralis)

• PointConstraints VertexConstraints

91
RunTime Rigs to Non-RunTime
92
Non-RunTime Rigs

• Helper Joints Definition
• They can be controlled by SetDrivenKeys,
  Expressions, Simple Rigs (constraint system), or
  Super Advanced Rigs (really any simulation you
  can think of) that are baked-out.

93
Non-RunTime Rigs - Baked Out

• BakeOut examples
• Bicep
• Complex Helper Joints constrained to nurbsMuscle
  influence object, baked out, 1 key per frame,
  scene cleaned.

94
Non-RunTime Rigs - Baked Out

• BakeOut examples
• Pectorals
• Complex Helper Joints constrained to nurbsMuscle
  influence object, baked out, 1 key per frame,
  scene cleaned.

95
Non-RunTime Rigs - Baked Out

• BakeOut examples
• Pectorals
• Lots!

96
Non-RunTime Rigs - SplineIK

• Great Skin-sliding effect

97
Non-RunTime Rigs - Jiggle

• Basic Jiggle is fairly easy to create in Maya,
  either with a
• Simple Expression
• frameCache node
• Could be turned into RunTime if your code
  supported it

98
Non-RunTime Rigs - Jiggle

• Create a 1 (or more) frame lag in global space
  for a joint
• BaseJoint Goal
• Blue joint parented to spine
• EndJoint Weighted
• Yellow joint lagging behind Blue joint by 1 frame

99
Non-RunTime Rigs - Jiggle

• Expression
• int frame frame // get the current frame
• float tx_0 BellyJiggleBase.translateX
• float tx_1
• // reset the attributes on start frame
• if (frame 1) // start frame
• tx_1 tx_0
• BellyJiggleEnd.translateX tx_0
• else
• BellyJiggleEnd.translateX tx_1
• tx_1 tx_0
• Repeat for Y Z axis

100
Non-RunTime Rigs - Jiggle

• frameCache Node

101
Non-RunTime Rigs - Jiggle
frameCache Expression
102
Non-RunTime Rigs - Jiggle

• Incorporation
• Could theoretically add a jiggle joint as a child
  of the helper joints
• Then just weight individual vertices to these
  jiggle joints to cause jiggling
• Or just make separate jiggle joints and their
  base joint wherever needed
• Where are they needed?

103
Non-RunTime Rigs to Automation
104
Automating Production

• Scripts
• Workflow
• Offline Muscle Systems
• Procedurally generated Helper Joints

105
Automating Production

• Scripts Tools youll need
• Script to auto-build entire Helper Joint System
  on your character
• Selectable body parts
• Scripts to automate editing of Rigs,
  SetDrivenKeys, Expressions for the artists
• Scripts to do lots of mirroring
• Helper Joint Rigs
• Weighting
• Set Driven Keys
• Export and Import Set Driven Keys
• Hint look at the animCurves and add custom attrs
  for connections to them upon export
• Demo

106
Automating Production

• Workflow
• Duplicate your primary skeleton and skin
• Apply your muscle system to this duplicate
  character
• Constrain duplicate skeleton to primary skeleton
• Hide duplicated skeleton, muscles, rigs.
  Template duplicated skin.

107
Automating Production

• Workflow

108
The Ultimate Reference

• Muscle Systems

109
The Ultimate Reference

• Muscle Systems

110
The Ultimate Reference

• Muscle Systems
• Auto imports muscle system
• Scaling to your skeleton
• AutoFits muscles to duplicated skin
• AutoSkins muscles to duped skin

111
The Ultimate Reference

• Once you have the Ultimate Reference, you now
  have the data to let a programmer put you out of
  a job. All (s)he needs to figure out is
• How many joints?
• Where to place them?
• How do they need to transform?
• What does their weighting need to be?

112
Mohr/Gleicher Method

• Siggraph 2003 Building Efficient, Accurate
  Character Skins from Example
• White Paper on auto-computing placement,
  animation, and weighting of Helper Joints.

113
Conclusion

• Weve covered just a few examples of how to use
  SetDrivenKeys, Constraints, and Expressions.
• My solutions are just first shot attempts. With
  refinement, you can come up with much better
  solutions.
• With these simple tools you can use them in many
  combinations to achieve nearly any effect you
  need.
• Of course everything is RunTime code dependent.
  As Character T.D.s we need to fight for
• SetDrivenKeys
• Constraints
• Point, orient, !!AIM!!, parent, geometry
• Expressions
• Spline IK
• frameCache (jiggle)
• Helper Joints are just a temporary solution until
  we can get more advanced deformers at RunTime to
  simulate Muscles
• Per-vertexDeformer (influence objects)
• WrapDeformer (lattices)
• fusiForm deformer
• . . . Talk more about this next year?

114
Thanks

• for your time.
• Please fill out surveys so I can make a better
  one next year.
• Peace out

115
Contacts, Reference, Credits

• Email
• jason_at_jason-parks.com
• jason_parks_at_playstation.sony.com
• Webpage (Helper Joints-GDC 2005)
• http//www.jason-parks.com/HelperJoints