Chris Oat

1
Advanced Character Rendering

• Chris Oat
• ATI Research, Inc.

2
Outline

• Skin Rendering
• Algorithm Overview
• Texture-space lighting
• Blur and dilation
• Soft shadows
• Translucent Shadows
• Acceleration technique Early-Z culling
  optimization
• Hair Rendering
• Modeling/Texturing
• Shading
• Depth sorting Early-Z culling optimization

3
Skin Rendering

• The human eye is very sensitive to the perception
  of skin, particularly with respect to facial
  recognition
• Most of the lighting from skin comes from
  subsurface scattering
• Skin color is mainly from the epidermis
• Pink/Red color mainly from blood in the dermis
• Traditional Lambertian lighting model assumes
  hard surfaces with no subsurface scattering so it
  doesnt work well for skin

4
Approximate Skin Cross Section
5
Our Objective

• Develop a simple, efficient skin rendering
  algorithm for real-time applications
• Must be very fast as it is only one of several
  rendering techniques that will be used
  concurrently to render realistic characters
• Results should approximate subsurface scattering

6
Basis for Our Approach

• SIGGRAPH 2003 sketch Realistic Human Face
  Rendering for The Matrix Reloaded
• Rendered a 2D light map
• Simulate subsurface diffusion in image domain
  (different for each color component)
• Used traditional ray tracing for areas where
  light can pass all the way through (e.g.. Ears)

7
Texture-Space Lighting

• Realistic Human Face Rendering for The Matrix
  Reloaded (SIGGRAPH03)
• Our results

8
Real Time Texture-Space Lighting

• Render diffuse lighting into an off-screen
  texture using texture coordinates as positions
• Blur the off-screen diffuse lighting
• Read the texture back and add specular lighting
  in subsequent pass
• Bump map is used only for specular lighting pass
  (high frequency detail is lost due to diffusion)

9
Texture Coordinate as Position

• Light as a 3D model but draw into texture
• Vertex shader outputs texture coordinates as
  projected position then the rasterizer does the
  unwrap
• Vertex shader computes light vectors based on 3D
  position and interpolates

10
Basic Approach
11
Texture-Space Lighting Vertex Shader
VsOutput main (VsInput i) // Compute output
texel position in screen space VsOutput o
o.vPos.xy i.texCoord2.0-1.0// move from 0,1
to -1,1 o.vPos.zw 1.0 // Pass along
texture coordinates o.texCoord i.texCoord
// Skin (Transform vertex from object space to
world space) float4x4 mSkinning
SiComputeSkinningMatrix(i.weights,
i.indices)
float4 vPos mul(i.vPos, mSkinning)
o.vNormal mul(i.vNormal, mSkinning) //
Perspective divide needed for shadow mapping
vPos vPos/vPos.w // Compute light vectors
using world space vertex // position . . .
12
Texture-Space Lighting Pixel Shader
float4 main (PsInput i) COLOR // Compute
Diffuse for Light 0 float3 vNormal
normalize(i.vNormal) float3 cLightColor
SiGetLightColor(0) float3 vLight
normalize(i.vLightVec0) float3 cDiffuse
dot(vNormal, vLight) cLightColor
// Compute Diffuse for Light 1 2 . . .
// Look up blur size and save in alpha float
sBlurSize tex2D(tBlurSize, i.vTexCoord)
float4 o float4(cDiffuse, sBlurSize) return
o
13
Spatially Varying Blur

• Blur to simulate the subsurface scattering
  component of skin lighting
• Use a grow-able Poisson disc filter
• Read the kernel size from a texture
• Allows varying the subsurface effect
• Higher for places like ears/nose
• Lower for places like cheeks

14
Filter Kernel

• Stochastic sampling
• Poisson distribution
• Samples stored as 2D offsets from center

15
Blur Kernel Size Map
16
Dilation

• Texture seams can be a problem (unused
  texels, bilinear blending artifacts)
• During the blur pass we need to dilate
• Use alpha channel of off-screen texture
• If any sample has 1.0 alpha, just copy the sample
  with the lowest alpha

17
Dilation Results

• Without Dilation

With Dilation
18
Shadows

• Shadow mapping
• Apply shadows during texture lighting pass
• Get free blur
• Soft Shadows
• Simulates subsurface interactions
• Lower precision/size requirements
• Reduces shadow map artifacts
• If using multiple lights, its still the same
  number of blur passes

19
Shadow Maps

• Create projection matrix to generate depth map
  from the lights point of view
• Use bounding sphere of head/character to ensure
  texture space is used efficiently
• Shadow map rendering pass Write depth (distance
  from light) into off-screen texture
• On texture lighting pass Test depth values in
  texture-space lighting pixel shader

20
Basic Approach Shadows
21
Shadow Map Vertex Shader
float4x4 mSiLightProjection // Light projection
matrix VsOutput main (VsInput i) VsOutput
o // Compose skinning matrix float4x4
mSkinning SiComputeSkinningMatrix(i.weights,

i.indices) // Skin position/normal and
multiply by light matrix float4 vPos
mul(i.vPos, mSkinning) o.vPos mul(vPos,
mSiLightProjection) // Compute depth (Pixel
Shader is just pass through) float dv
o.vPos.z/o.vPos.w o.depth float4(dv, dv,
dv, 1) return o
22
Texture Lighting with Shadows Vertex Shader
VsOutput main (VsInput i) // Same lead in
code as before . . . // Compute texture
coordintates for shadow map o.vLightPos
mul(vPos, mSiShadowLight) o.vLightPos /
o.vLightPos.w o.vLightPos.xy
(o.vLightPos.xy 1.0f)/2.0f o.vLightPos.y
1.0f-o.vLightPos.y o.vLightPos.z - 0.01f
return o
23
Texture Lighting with Shadows Pixel Shader
sampler tShadowMap float sFaceShadowFactor float
4 main (PsInput i) COLOR // Same lead in
code . . . // Compute Light 0 float3
cLightColor SiGetLightColor(0) float3
vLight normalize(i.vLightVec0) float NdotL
SiDot3Clamp(vNormal, vLight) float VdotL
SiDot3Clamp(-vLight, vView) float4 t
tex2D(tShadowMap, i.vLightPos.xy) float lfac
sFaceShadowFactor if (i.vLightPos.z lt t.z)
lfac 1.0f float3 cDiffuse lfac
saturate((fresnelVdotLNdotL)lightColor
) // The rest of the shader is the same as
before . . .
24
Shadow Map Shadowed/Lit Texture
Shadow Map (depth)
Shadows in Texture Space
25
(No Transcript)
26
Shadow from Translucent Objects

• Algorithm
• Draw depth of opaque shadow geometry to shadow
  buffers alpha channel
• Additively blend RGB of translucent shadow
  geometry into shadow buffer RGB channels
• Depth test is ON
• Depth write is OFF
• Texture-space lighting pixel shader must now
  blend non-shadowed pixels by the translucent RGB
  value

27
Basic Approach Translucent Shadows
28
Shadow Map for Transparent Shadows
Translucent shadow map
Opaque shadow map
RGB
29
Translucent Shadow Results
Translucent Shadows
Opaque Shadows
30
Specular Lighting

• Use a bump map for specular lighting
• Per-pixel exponent
• Need to shadow specular light
• Bright specular light in heavily shadowed regions
  wouldnt make sense!
• Too expensive to do another blur pass shadows
• Use luminance of blurred diffuse texture
• Modulate specular from shadowing light by
  luminance of texture space light (very low
  frequency)
• Darkens specular in shadowed areas but preserves
  specular in unshadowed areas
• Not a limitation, just an optimization
• You could do a separate blur pass for shadows
  (rather than bake them into diffuse light texture)

31
Normal Map Compression

• Exposed in Direct3D using FOURCC ATI2
• Two channels (x and y)
• Each channel is compressed separately
• Each uses a block encoding like DXT5 alpha
• Derive z in pixel shader
• Useful for tangent space normal maps (where the
  sign of the z component is known)
• Can be used for any two channel texture

32
Final Pixel Shader (with Specular)
sampler tBase sampler tBump sampler
tTextureLit float4 vBumpScale float4 main
(PsInput i) COLOR // Get base and bump
map float4 cBase tex2D(tBase,
i.texCoord.xy) float3 cBump tex2D(tBump,
i.texCoord.xy) // Get bumped normal
float3 vNormal SiConvertColorToVector(cBump)
vNormal.z vNormal.z vBumpScale.x vNormal
normalize(vNormal)
33
Final Pixel Shader (with Specular) Continued
// View, reflection, and specular exponent
float3 vView normalize(i.vView) float3
vReflect SiReflect(vView, vNormal) float
exponent cBase.avBumpScale.z
vBumpScale.w // Get "subsurface" light
from lit texture. float2 iTx i.texCoord.xy
iTx.y 1-i.texCoord.y float4 cLight
tex2D(tTextureLit, iTx) float3 cDiffuse
cLightcBase
34
Final Pixel Shader (with Specular) Continued
// Compute Light 0 float3 cLightColor
SiGetLightColor(0) float3 vLight
normalize(i.vLight0) float RdotL
saturate(dot(vReflect, vLight)) float shadow
SiGetLuminance(cLight.rgb) shadow
pow(shadow,2) float3 cSpecular
saturate(pow(RdotL,exponent)lightColor)
cSpecular shadow // Compute Light 1 2
(same as above but no shadow term) . . .
// Final color float4 o o.rgb cDiffuse
cSpecular o.a 1.0 return o
35
Specular Shadow Dimming Results
Specular Without Shadows
Specular With Shadows
36
Using Early-Z for Culling

• Testing z-buffer prior to pixel shader execution
• Can cull expensive pixel shaders
• Only applicable when pixel shader does not output
  depth
• This texture-space operation doesnt need the
  z-buffer
• We could store data other than depth in z-buffer
• Use Early-Z to cull useless computations
• Back face culling
• Distance and frustum culling
• Set z-buffer on lighting pass according to
  frustum, distance from viewer and facing-ness of
  polygons
• Set z-test such that non-visible polygons fail
  z-test
• Reduces cost of image-space blur in regions that
  dont need it (because they arent visible)

37
Back Face Culling
Back facing pixels culled using early-z
Over the shoulder view of Ruby
38
Skin Rendering Summary

• Simple and fast skin rendering algorithm
• Texture-space blur with dilation
• Soft shadows for free
• Translucent shadows
• Early-Z culling acceleration techniques

39
Demo
40
Skin Rendering References

• Borshukov03 Borshukov and Lewis. Realistic
  Human Face Rendering for The Matrix Reloaded.
  Technical Sketches, SIGGRAPH 2003.
• Mertens03 Mertens et al. Efficient Rendering of
  Local Subsurface Scattering. Pacific Graphics
  2003.

41
Hair Rendering Overview

• Hair rendering technique using polygonal hair
  model
• Shading Mix of
• Kajiya-Kay hair shading model
• Marschners model presented at SIGGRAPH 2003
• Simple, approximate depth-sorting scheme
• Early-Z culling optimizations

42
Hair Rendering

• Hair is important visually
• Most humans have some hair on their heads
• Hair is challenging to render
• There is a lot of it! (100k 150k strands on a
  human head)
• Many different hair styles
• 25 of the total render time of Final Fantasy
  The Spirits Within was spent on the main
  characters hair

43
Hair Model - Geometry

• Several layers of patches to approximate
  volumetric qualities of hair
• Per-vertex ambient occlusion term to approximate
  self-shadowing

44
Hair Model Geometry

• Reasons for using a polygonal hair model
• Lower geometric complexity than line rendering
• Makes depth sorting easier/faster
• Pretty much a necessity for real time use on
  current graphics hardware
• Integrates well into current art pipelines

45
Hair Model - Textures

• Base texture
• Stretch noise
• Hair color set in a shader constant
• Alpha texture
• Should have fully opaque regions
• Specular shift texture
• Specular noise texture

Base Texture
Alpha Texture
46
Hair Lighting Kajiya-Kay

• Anisotropic strand lighting model
• Use hair strand tangent T instead of normal N in
  lighting equations
• Assumes hair normal to lie in plane spanned by T
  and view vector V
• Example Specular NH term

47
Hair Lighting Marschner

• Based on measurements of hair scattering
  properties
• Observations
• Primary specular highlight shifted towards hair
  tip
• Secondary specular highlight
• Colored
• Shifted towards hair root
• Sparkling appearance
• For simplicity were trying to match these
  observations phenomenologically

Primary highlight
Secondary highlight
Hair strand interior
48
Hair Shader Implementation

• Vertex shader
• Just passes down tangent, normal, view vector,
  light vector and ambient occlusion term
• Pixel shader
• Diffuse lighting
• Two shifted specular highlights
• Combines lighting terms

49
Diffuse Lighting Term

• Kajiya-Kay diffuse term sin(T, L) looks too
  bright without proper self-shadowing
• Instead, use scaled and biased NL term
• Brightens up areas facing away from the light
  when compared to plain NL term
• Simple subsurface scattering approximation
• Softer look

50
Shifting Specular Highlights

• Move tangent along patch normal direction to
  shift specular highlight along hair strand
• Assuming T is pointing from root to tip
• Positive shift moves highlight toward tip
• Negative shift moves highlight toward root
• Look up shift value from texture to break up
  uniform look over hair patches

float3 ShiftTangent (float3 T, float3 N,
float shift) float3 shiftedT T
shift N return normalize(shiftedT)
51
Specular Strand Lighting

• Specular strand lighting using half-angle vector
• Using reflection vector and view vector would
  make the shader a little more complicated
• Two highlights with different colors, specular
  exponents and differently shifted tangents
• Modulate secondary highlightwith noise texture

float StrandSpecular (float3 T, float3 V,
float3 L, float exponent)
float3 H normalize(L V) float dotTH
dot(T, H) float sinTH sqrt(1.0 -
dotTHdotTH) float dirAtten
smoothstep(-1.0, 0.0,
dot(T, H)) return dirAtten pow(sinTH,
exponent)
52
Putting it All Together
(Note external constants are orange)
float4 HairLighting (float3 tangent, float3
normal, float3 lightVec,
float3 viewVec, float2 uv, float ambOcc) //
shift tangents float shiftTex
tex2D(tSpecShift, uv) 0.5 float3 t1
ShiftTangent(tangent, normal, primaryShift
shiftTex) float3 t2 ShiftTangent(tangent,
normal, secondaryShift shiftTex) // diffuse
lighting the lerp shifts the shadow boundary for
a softer look float3 diffuse
saturate(lerp(0.25, 1.0, dot(normal, lightVec))
diffuse diffuseColor // specular
lighting float3 specular specularColor1
StrandSpecular(t1, viewVec, lightVec,

specExp1) // add 2nd specular term, modulated
with noise texture float specMask
tex2D(tSpecMask, uv) // approximate sparkles
using texture specular specularColor2
specMask StrandSpecular(t2, vieVec, lightVec,

specExp2) // final color assembly
float4 o o.rgb (diffuse specular)
tex2D(tBase, uv) lightColor o.rgb ambOcc
// modulate color by ambient occlusion
term o.a tex2D(tAlpha, uv) // read alpha
texture return o
53
Combination of Lighting Terms
54
Comparison
55
Approximate Depth Sorting

• Back-to-Front rendering order necessary for
  correct alpha-blending
• For a head with hair this is very similar to
  rendering from inside to outside
• Use static index buffer with inside to outside
  draw order
• Computed a preprocess time
• Sort connected components (hair strand patches)
  instead of individual triangles

56
Sorted Hair Rendering Scheme

• Pass 1 opaque parts
• Enable alpha test to only pass opaque pixels
• Disable backface culling
• Enable Z writes, set Z test to Less
• Pass 2 transparent back-facing parts
• Enable alpha test to pass all non-opaque pixels
• Cull front-facing polygons
• Disable Z writes, set Z test to Less
• Pass 3 transparent front-facing parts
• Enable alpha test to pass all non-opaque pixels
• Cull back-facing polygons
• Enable Z writes, set Z test to Less

57
Performance Tuning

• Use early Z culling extensively to save us from
  running expensive pixel shader
• Usually half the hair is hidden behind the head
• Draw head first
• Early Z culling cant be used when alpha test is
  enabled!
• Solution Prime Z buffer with a very simple
  shader that uses alpha test
• Use Z testing instead of alpha testing in
  subsequent passes for same effect
• Early Z culling saves considerable fill overhead!

58
Optimized Scheme Part 1

• Prime Z-Buffer with depth of opaque hair regions
• Enable alpha test to only pass opaque pixels
• Disable backface culling
• Enable Z writes , set Z test to LESS
• Disable color buffer writes
• Use simple pixel shader that only returns alpha
• No benefits from early-Z culling in this pass,
  but shader is very cheap anyway

59
Optimized Scheme Part 2

• Render opaque regions
• Start using full hair pixel shader
• Disable backface culling
• Disable Z writes
• Set Z test to EQUAL
• Z test passes only for fragments that wrote to Z
  in pass 1, which are the opaque regions
• This and subsequent passes dont require alpha
  testing and thus benefit from early-Z culling

60
Optimized Scheme Part 3

• Render transparent back-facing parts
• Cull front-facing polygons
• Disable Z writes
• Z order isnt necessarily correct
• Set Z test to LESS

61
Optimizing Scheme Pass 4

• Render transparent front-facing parts
• Cull back-facing polygons
• Enable Z writes
• Set Z test to LESS
• Enabling Z writes prevents incorrect depth order
  at expense of possibly culling too much
• Covers up potential depth order artifacts of
  previous pass

62
Demo
63
Pros and Cons Of This Approach

• Pros
• Low geometric complexity
• Reduced load on vertex engine
• Makes depth sorting faster
• Easy fall-back for lower-end hardware
• Cons
• Sorting scheme assumes little to no animation in
  hair model
• Pony tails need to be handled seperately
• You could sort at runtime to overcome this
• Not suitable for all hair styles

64
Hair Rendering Summary

• Polygonal hair model
• Hair lighting
• Simple approximate depth-sorting scheme
• Optimization Tips

65
Hair Rendering References

• J. Kajiya and T. Kay. Rendering fur with three
  dimensional textures. In SIGGRAPH 89 Conference
  Proceedings, pp. 271-280, 1989.
• Stephen R. Marschner, Henrik Wann Jensen, Mike
  Cammarano, Steve Worley, and Pat Hanrahan, Light
  Scattering from Human Hair Fibers. In Proceedings
  of SIGGRAPH 2003.
• SIGGRAPH 2003 Hair Rendering Course Notes

66
Putting it All TogetherRuby The Double Cross
67
Summary

• Efficient and Scalable techniques for rendering
  characters
• Texture-space lighting for simulating subsurface
  scattering in skin
• Polygonal hair strand modeling and rendering
• Early-Z optimizations

68
Thank you!

• Dave Gosselin
• Thorsten Scheuermann
• Pedro Sander
• Chris Brennan

69
Questions?
Chris Oat coat_at_ati.com