GDC 2005

1
(No Transcript)
2
(No Transcript)
3
Notes on downloaded slides

• The accompanying videos can be downloaded from
• www.ChrisEvans3D.com/tutorials.htm
• The videos were captured with various software
  which sometimes seriously butchers the framerate,
  please keep this in mind
• We have added comments generally summing up what
  we said about each slide
• The audio of our session in mp3 format can be
  purchased from GDC Radio (http//store.cmpgame.c
  om/product.php?id1847cat)

4
Christopher EvansSenior Technical Artist, Crytek

• Responsibilities
• RD Artist
• Artist tools
• Getting characters in-game

5
Hanno HagedornSenior Character Artist, Crytek

• Responsibilities
• Character Heads
• Modeling
• Texturing
• Rigging
• Facial Setup
• Character Bodies (partially)

6
What is this stuff?

• A cross section of current character-related RD
  at Crytek
• Mostly WIP, all real-time, real assets
• Presenting collaborative work
• Artist-driven, or real world data-driven tools

7
Intended Audience

• Intermediate to Advanced Requires experience
  and familiarity with the subject
• Technical Artists (TA) and artists with technical
  interests
• Not too complicated
• Simple, pragmatic approaches

8
Session Overview

• Low Level
• Deformation
• How you drive deformations
• How you deform geometry
• Animation
• High Level
• Character Variation
• Case Studies
• Internal Tools
• Asset Creation
• Facial Editor
• Setup
• Animation

9
What is a Character?
10
What is a Character?
Rejected internal concept art

• Person or creature?

11
What is a Character?
Rejected internal concept art

• Vehicle or machine with
• moving parts?

12
What is a Character?

• Any object or set of objects whose movement is
  derived from a skeleton.

13
What is a Character?

• People and creatures
• Vehicles and machines
• Moving character attachments
• Guns, backpacks, heads, hands
• Dynamic, jointed physics simulations
• Ropes, complex constraint setups
• Pre-baked physics and broken objects
• Pre-baking cinematic destruction

14
VIDEO 0004
15
The Low LevelDeformation
George Bridgman, The Human Machine 1929
16
The Low LevelDeformation

• What drives the deformation?
• Motion?
• Arbitrary inputs?
• Shader-based?
• How do you move and deform geometry?
• Is it efficient?
• What kind of approach?

17
VIDEO 0045
VIDEO 0045
18
The Low LevelDeformation

• What drives the deformation?
• Motion?
• Arbitrary inputs?
• Shader-based? (procedural noise)
• How do you move and deform geometry?
• Is it efficient?
• What kind of approach?

19
Driving DeformationPose Driven Systems
George Bridgman, The Human Machine 1929
20
Driving deformation Pose-Driven Systems

• You can drive many things with skeletal poses
• Morph Targets
• Joints
• Maps
• Outside our industry Not a new idea at all

21
Simple 3 Vector Driver Shape
VIDEO 0103
22
Simple Driver Color and Shape
VIDEO 0109
23
Driving deformation Pose-Driven Systems

• Pre-baked
• Baked to keys/shapes
• Runtime
• In engine

24
Pros/Cons Pre-Baked Pose-Drivers

• Pros
• Setup is in the asset creation app
• Implementations work well in most animation-based
  games (joints)
• Cons
• Non-dynamic - Does not work with motion generated
  at runtime
• Ragdoll physics
• Physically-based motion
• Other procedurally-generated motions

25
Driving deformation Pose-Driven Systems

• Pre-baked
• Baked to keys/shapes
• Runtime
• In engine

26
Driving deformationRuntime Pose-Drivers

• Pros
• Works dynamically in any situation
• Only limit is the number of poses you define
• Cons
• Difficult to set up
• Need to mirror in the asset package
• Constraints
• Parameterization

27
Driving deformationArbitrary-Drivers

• Holy grail of driven systems
• General Arbitrary Drivers
• General input/output system
• Houdini, Shadetree, etc
• The more general the better
• Runtime expressions?
• Not limited to deformation at all!

28
VIDEO 0116
29
VIDEO 0211
30
VIDEO 0254
31
VIDEO 0319
32
The Low LevelDeformation

• What drives the deformation?
• Character motion? Skeletal motion?
• Arbitrary inputs?
• Shader-based? (procedural noise)
• How do you move and deform geometry?
• Is it efficient?
• Is it an Artist-centric approach?

33
DeformationSkinning
George Bridgman, The Human Machine 1929
34
Linear Skinning Weight vertices in a linear
fashion to n bones

• Definitely nothing new
• First use in a game (software)
• Trespasser? (1997-1998)
• First use in a game (hardware)
• dx8 via vert shaders
• dx7 rarity
• Hardware deceleration

35
Linear Skinning,or smooth skinning, or linear
blend skinning..

• Pros
• Industry Standard
• Lots of available research
• Cons
• Poor volume preservation
• The more a joint bends, the more it collapses
• Difficult to get specific control
• Can be more resource intensive

36
Our Implementation ofSpherical Skinning

• Hardware Skinning with Quaternions (2004)
• by Jim Hejl (EA) Game Programming Gems 4
• First game to utilize it?

37
Pros/Cons Spherical Skinning

• Pros
• Much better volume preservation
• Less resource usage
• (Skinning position, tangent, and bi-normal)
• 72 bones per draw call
• 48 (Linear w/ matrices)
• 72 (Linear w/ quats)
• 20 instructions
• 40 instructions (Linear w/ matrices)
• 60 instructions (Linear w/ quats)

38
VIDEO 0411
39
VIDEO 0445
40
Pros/Cons Spherical Skinning

• Cons
• Needs support in the asset creation package
• Plugin, preview, etc
• Inherent issues with some weighting
  configurations
• N bones per vert
• Non-consecutive or interleaved bones

41
Interleaved BonesExample Character Studio Biped
42
Pros/Cons Spherical Skinning

• Cons
• Needs support in the asset creation package
• Plugin, preview, etc
• Inherent issues with some weighting
  configurations
• N bones per vert
• Non-consecutive or interleaved bones
• The volume preservation sometimes corrects things
  we have come to rely on linear skinning for

43
Interested in New Skinning Technology?

• GDC Session Skinning with Dual
• Quaternions
• Tomorrow (Fri)
• 9am 10am
• Room 2009, West Hall

44
DeformationMorph Targets
George Bridgman, The Human Machine 1929
45
DeformationMorphs are Your Friends

• Also known as
• Blendshapes, shapes, vertex animation
• Artistic approach Artist defines the look
  completely
• Facial animation
• Sculpt muscles, phonemes, visimes
• Corrective shapes
• Driven by arbitrary inputs
• Baked simulations
• Bending metal, ocean waves, soft tissue

46
DeformationMorphs are Your Friends

• Keep them light in engine
• Sparse morphs
• Only the verts that move
• Corrective shapes can be very light
• Only move the points that you have to
• Sparse morphs on DX10 hardware
• Sparse morphs running on dx10 hardware
• Supports binding to more than 4 color images
• Store morphs in a texture
• Blend between textures
• There is a demo in the DX10 SDK
• They also have wrinkle maps in this example!

47
Driving Morphs

• Pose-Driven morphs
• Artist-Sculpted Pipeline
• Define pose vectors
• Artist sculpts the pose
• Link sculpts to those pose vectors
• Parameterize/blend between poses
• Simulation-Sculpted
• Artist creates deformation
• Links deformation to some arbitrary parameter in
  the engine

48
Driving Morphs

• Our asset creation tool
• Allows the artist to pose the rigged character
• We decided to use pre-defined pose vectors
• Generates a mesh in that pose for him to sculpt
• Extracts the artists sculpt with the skin
  deformation culled
• Created a shape for that pose

49
VIDEO 0526
50
VIDEO 0643
51

• Our In Engine Setup
• Apply the shape locally before the skinning
• Assign the shape to the correct bone and pose
  vector
• This is the simplest scenario
• Single pose vector

52
VIDEO 0649
53
This is a simple test, there are many more uses!

• Corrective morph shapes
• SAY GOODBYE TO SKINNING ARTIFACTS!
• Complex deformation
• You dont have to sculpt these targets by hand!
• The sky is the limit

54
VIDEO 0914
55
VIDEO 1035
56
DeformationHelper Joints
George Bridgman, The Human Machine 1929
57
Helper Joints

• Jason Parks Advanced Deformations on Runtime
  Characters GDC 05/06
• http//www.jason-parks.com/HelperJoints/
• http//www.jason-parks.com/MuscleSystems/
• Great but less artist-driven
• Universal
• Needs a lot of technical support
• Non-intuitive
• Examples
• Same driving issues as with morphs
• Pre-baked vs. Runtime

58
VIDEO 1046
59
Example Shark
60
ExampleNanosuit
61
ExampleNanosuit
62
ExampleNanosuit

• Helper Joints (12)
• Deltoids (shoulders) (8 joints)
• 4 on each shoulder
• Pectoralis (chest) (2 joints)
• 1 per side
• Latissimus (back) (2 joints)
• 1 per side

63
VIDEO 1120
64
All these systems, when do I use what?

• Dependent
• On your pipeline
• Result you want to achieve

65
The Low LevelAnimation
George Bridgman, The Human Machine 1929
66
The Low Level Animation

• IK Retargetting
• Look IK
• Analytical IK for arms/legs
• Feet on uneven surfaces
• CCD IK
• 7 DoF for bone chains
• Used for retargeting and planting
• Parameterization IK can yield good results
• IK fallback for dynamic motion

67
VIDEO 1208
68
Character Variation
69
Character Variation

• System supports 3 types of variation
• Morph-based
• Attachment-based
• Decal-based

70
Character Variation

• Morph-based
• Thinner/fatter body types
• Change size/position of design elements
• Characters divided up into regions to mask
  morph variations (vertex colors)
• Attachment-based
• Skin Attachments
• Object Attachments
• Decal-based
• Small 128px dirt, scars, camouflage, etc
• Layers on top of current texture
• Super low resource usage

71
Case StudyVariations for E3 2006

• 3 x 128px Camouflage Face Textures
• 1 x 128px Dirt Texture
• 2 Head Variations
• 2 Body Variations
• Thinner/Fatter body types, Longer/Shorter
  clothing
• Attachments
• Skin Attachments
• Hands, Heads
• Object Attachments
• Pouches, Helmets, Ammunition

72
E3 Character Variations 1/10
73
E3 Character Variations 2/10
74
E3 Character Variations 3/10
75
E3 Character Variations 4/10
76
E3 Character Variations 5/10
77
E3 Character Variations 6/10
78
E3 Character Variations 7/10
79
E3 Character Variations 8/10
80
E3 Character Variations 9/10
81
E3 Character Variations 10/10
82
Issues We Ran Into

• Morph-based
• Maintaining point index in LODs (variations)
• Too Much Detail? Not enough?
• Designer setup time
• Procedural runtime generation (examples-based)
• How much of a difference can the player perceive
  ingame?
• Parameterization
• Non-Face attachments
• Eye placement
• Bone attachments

83
Case StudyCharacter Flash Freezing

• Initial Freezing
• Must be dynamic
• Any character in any pose or animation
• Slowly blend, do not pop to a stop
• Broken Pieces
• Physicalized parts
• Interactive, physicalized parts
• Realistic
• Actual parts

84
Case StudyCharacter Flash Freezing

• Implementation
• Frozen Mesh
• Procedural Ice Shader (layer)
• Brittle Physicalized mesh
• Shattering
• Ice particles emitted from joint phys proxies
• Broken human bodyparts are created for most of
  the bones in the character.
• Pieces are spawned in the orientation of their
  associated joints
• Broken pieces
• Left as individual physicalized entities
• Pickable

85
VIDEO 1312
86
Baking Out Simulations Isnt That Cheating?
87
Baking Out Simulations Isnt That Cheating?

• Same thrilling game experience for mid-low end
  users (though less interactive)
• Benefits
• Musculo-skeletal systems/Organic Deformation
• Bake to joint-driven morphs
• Bake to joint-driven joints
• Rigid-body physics
• Bake rigid body physics to keyframe data
• 1 joint per piece
• Baked but still interactive?

88
Baking Out Simulations Interactivity

• Organics
• Pose-driven joints or morphs
• Physicalization of Rigid Bodies
• Use the keyframe data to generate accurrate
  impulse
• Physicalize the animated rigid body upon
  collision
• Physicalize after the animation plays
• Physicalize after the joint has reached x
  distance from the object center.
• Physicalize when received damage

89

• Pre-Baked Rigid Bodies

VIDEO 1400
90
Baking Out Simulations Characters at Heart

• Keyframe data
• Parameterization
• Blend between multiple simulations
• Inherent issues, but for low spec much better
  than the alternative
• Compression
• Can have attachments
• Deformible (bending metal etc)
• Can attach other characters to it

91
Baking Out Simulations Issues We Ran Into

• Draw calls
• Meshes need to be merged into one
• Compression
• Piles of debris
• Export surroundings for external sim
• Wavefront OBJ export
• Terrain, objects, etc
• Can run the simulation in Max, Maya, etc
• Better animator control for Hollywood 11

92
VIDEO 1417
93
Hanno HagedornSenior Character Artist, Crytek

• Responsibilities
• Character Heads
• Modeling
• Texturing
• Rigging
• Facial Setup
• Character Bodies (partially)

94
I am going to talk about...

• Pipeline improvements
• Polybump2
• How we do faces, from the texture to animation
• Asset Creation
• Textures
• Model
• Shaders
• Facial Animation System

95
Crytek PolyBump2

• Crunches assets with many millions of polygons
• Capable of combining multiple models, each with
  many millions of polygons into one single
  calculation
• 32/64bit 3dsmax plugin or standalone application
• Exports normal maps, displacement maps, and
  ambient occlusion maps
• Gives out a storage file containing 100x the
  accuracy of a normal texture map
• Extracting new data without recalculation

96
(No Transcript)
97
(No Transcript)
98
(No Transcript)
99
(No Transcript)
100
Unoccluded Area Direction

• Simulates self shadowing inside the normal map
  while softening hard details
• Combines ACC with the normalmap
• Softshadowing within the normalmap
• Applies a very soft look to the skin

101
VIDEO 1440
102
(No Transcript)
103
VIDEO 1458
104
Asset Creation

• Get good texture reference
• Bad texture reference
• Synchronizing texture maps
• Get the right software

105
Get Good Texture Reference

• We all love hand painted textures
• Hand painting photorealistic / Schedule pain
  in the ass
• Well lit photos are key
• Unwanted lighting will plague you later on
• The less lighting information will get you closer
  to your desired result

106
Bad texture reference

• Very colored
• Very blurry
• Shiny nose
• Lots of direct and only few ambient light
• Very contrasted shading

107
Good Texture reference

• Equaly lit
• Very sharp
• No major highlights
• Interesting details
• Small colorvariations in the skin

108
Synchronizing texture maps

• Model and texture should feel like a single unit
• Pore stamps can be very handy but
• Never fit the existing texture
• Create a repetitive look
• Make sure your diffuse and spec map complement
  your high frequency or pore-detailed model

109
Base Diffuse Texture
110
Normalmap
111
Specular Map
112
Diffuse with ACC lighting
113
Get the right Software

• Z-Brush and Mudbox
• Great programs for sketching out faces and shapes
• Great programs for polishing and detailing assets
• Silo is looking very promising
• Combining Modeling and sculpting into one workflow

114
Shader

• Rim Lighting
• Sub-Surface Scattering (SSS)
• Eyes

115
Rim Lighting

• Simulates greasy reflective Skin
• Works against the classic specular look of
  last-gen games
• There is no such thing as specular light in
  reality
• Similar to Fresnel reflections
• Adding rim lighting by
• Ambient color
• Cubemap
• Realtime automated cubemaps

116
No Rim Lighting
117
Basic Rim Lighting
118
Strong Rim Lighting
119
Sub-Surface Scattering (SSS)

• Approximates light rays penetrating the skin and
  exiting another spot
• Gives skin a nice reddish, fleshy look
• Even simulates thin vascular tissue
• SSS-map allows us to define the skins thickness
  and color

120
No SSS
121
Basic SSS
122
Strong SSS
123
Eyes - Windows to the Soul
Eyes - Windows to the Soul

• After several tests we decided to
• Build a physically correct Eye model with
  internal physiology
• Works better with new shader possibilities
• Special Iris shader glinting

3dsMax screenshot
124
Eyes - Windows to the Soul

• After several tests we decided to
• Build a physically correct Eye model with
  internal physiology
• Works better with new shader possibilities
• Special Iris shader glinting

3dsMax screenshot
125
Eyes - Self shadowing
Eyes - Windows to the Soul

• After several tests we decided to
• Build a physically correct Eye model with
  internal physiology
• Works better with new shader possibilities
• Special Iris shader glinting

3dsMax screenshot
126
Eyes - Self shadowing

• Using overlaying geometry for emulating the
  shaded eye
• Comments Eyelids and Eyeballs together
• This allows us to simulate the shadowing of the
  eyelashes and surrounding bone structure

3dsMax screenshot
127
Eyes - Self shadowing

• Self shadowing
• Using the same base as the Eyelids saves textures
  and draw calls
• Eyeball moves under the Overlaying Geometry while
  keeping the shading

3dsMax screenshot
128
Eyes - Self shadowing

• Self shadowing
• Using the same base as the Eyelids saves textures
  and draw calls
• Eyeball moves under the Overlaying Geometry while
  keeping the shading

3dsMax screenshot
129
Eyes - Self shadowing
VIDEO 1509
130
Eyes In motion

• Eyelids move along the Eyeball via Fleshy Eyelid
  morphs
• Automated Saccadic eye motion
• Automated Eye Blinking, synced with the Saccadic
  eye motion

131
Eyes In motion

• Eyelids move along the Eyeball via Fleshy Eyelid
  morphs
• Automated Saccadic eye motion
• Automated Eye Blinking, synced with the Saccadic
  eye motion

VIDEO1523
132
Facial Animation System
133
Facial Animation System

• Muscle Morphs
• Expressions
• Linear
• Non-linear
• Phonemes
• Visimes
• Joystick Controls
• Optical Flow

134
Morphs The Base

• Muscle Morphs
• 89 Morphs
• Each muscle a separate morph
• A facial expression is sculpted by muscle
  morphs
• Asymmetrical morph setup
• Natural face movement
• Adds character to the facial movement
• Automatic fleshy eye movement
• Eyelids move synchronized with the Eyeball
• Using same topology for all faces
• Easy transfer of morphs between meshes

135
Expressions The Connection

• Unlimited scalability
• Core system is very simple, yet very scalable
• Multiple linear and nonlinear expressions
• Can combine linear and non-linear
• Setting the base for the phoneme extraction

136
Linear Expressions

• Linear blending of morphs or drivers

VIDEO 1534
137
Non-linear expressions

• Non-linear blending of morphs
• Use multiple morphs to
• create a non-linear motion
• keep the volume of complex or twisted shapes

VIDEO1613
138
Expressions are drivers

• System drives Morphs, Bones, and attachments
• All elements are treated equally
• Can be easily combined or parameterized

139
Phonemes

• Phonemes
• The smallest distinctive units of speech, or the
  mouth shapes to make each sound
• Strong and weak phonemes for appropriate usage
• 96 phonemes, 48 strong and 48 weak
• Seamless blending between strong and weak phonemes

140
Phoneme

• Phonemes
• The smallest distinctive units of speech, or the
  mouth shapes to make each sound
• Strong and weak phonemes for appropriate usage
• 96 phonemes, 48 strong and 48 weak
• Seamless blending between strong and weak phonemes

141
Visimes

• Visual simplification of phonemes
• Phonemes like o, u, oo, and ow are
  visually very similar
• 14 visimes, 7 strong, 7 weak
• Massively simplifies the work with visimes
• Tweaking several phonemes at once is handy when
  transferring the setup to a different character

142
Visimes

• Visual simplification of phonemes
• In use by 2D animations for over 90 years

143

• Muscle/Phoneme/Visime Breakdown

144
Phoneme Extraction

• Automated process
• Automatically generates Phonemes from a pure
  Audio file
• Manual process
• Breaks words into phonemes and places them in the
  timeline corresponding to the audio waveform
• barely needs touching up, easy to handle,
  batch-able.

145
VIDEO 1715
146
Joystick Controls Manual input device

• User-friendly and intuitive
• Your Grandma could animate with this!
• Animators focus will be on animation and not
  dragging sliders back and forth

147
Joystick Controls Manual input device

• User-friendly and intuitive
• Your Grandma could animate with this!
• Animators focus will be on animation and not
  dragging sliders back and forth

148
Joystick Controls Manual input device

• Handles 4 diff expressions at once
• Example, Up/Down could be the jawdropper and
  jaw_presser, Left/Right could be the
  mouth_widener and mouth_tightener.
• Automatically excludes opposing morphs
• No more fighting with frown/smile or open/close
• Puppeting
• Real-time Mouse input capture

149
VIDEO 1728
150
VIDEO 1832
151
Joystick Controls scalability

• User-definable
• Joysticks plug into the cure interface, any
  expressions can be mapped to them
• Scalable toolset
• Can be used to limit the detail an animator sees
• You can predefine A, B, and C class joystick
  setups
• We got inspired by Jason Osipas book Stop
  Staring, which popularized joystick control
  setups

152
Optical-flow based Facial Motion Capture
153
Optical-flow based Facial Motion Capture

• Getting good results with a minimum amount of
  work
• No expensive hardware needed
• Animators here have used webcams at their desks.
• Very easy to polish
• MoCap system uses the same interface the
  animators do (joysticks)
• Combines with the auto phoneme extraction to
  polish lip-sync

154
Theoretical Marker Setup
155
Practical Marker Setup
156
Marker corresponding joysticks
157
Marker corresponding joysticks
VIDEO1942
158
The Capturing Workflow
VIDEO 1955
159
Final Result
VIDEO 2111
160
Final Result

• Casual B-Level animation
• Used a Hand camera for Motion capturing
• Used Eyeliner for the Markers
• Untouched Facial Mocap Data
• Automatic Phoneme Extraction
• Procedural Eye blinking
• Procedural Eye behavior

161
Final Result
VIDEO 2122
162
Upcoming Improvements

• Giving animations to the animation department
  for
• Enhancing and exagerating the acting
• Polishing up Mocap Data
• Cleaning up Mocap Data Keys
• Improved Phoneme Blending
• Improvement will come, but our mainfocus is on
  finishing up the game right now

163
Questions?
164
Questions or comments?Christopher Evans
Christopher_at_Crytek.de Chris.Evans_at_Gmail.comHann
o HagedormHanno_at_Crytek.de Hanno.Hagedorn_at_Polyroy
al.com For more info,come visit us at
boothXXXX