Technical Issues: Graphics

1
Technical IssuesGraphics
2
Technical IssuesGraphics

• A discussion of modern video games is incomplete
  without mention of graphics technology.
• Some game developers focus on these issues to
  such an extent that others suffer as a result.
• Good visual effects can make a good game great,
  but they will not make a game good.
• Nevertheless, they have become an important part
  of modern video games.

3
Technical IssuesGraphics
Screen shot from Unreal Tournament 2003. This
game is known for great graphics in its visual
effects. It is hard to imagine what itwould be
like without them.
4
Graphics Requirements for Video Games

• In essence, graphics are used to provide the
  representation element of a video game.
• Before such visuals, representation was provided
  by textual descriptions and a healthy dose of
  player imagination.
• In these cases, good storytelling allowed the
  player to create their own visuals in their heads.

5
Graphics Requirements for Video Games
Screen shot from the original Zork (left) and
Return to Zork (right), one of theearly PC
CD-ROM games. In the opening sequence of Return
to Zork, we finally get to visualize the
original white house. It was definitely a
totally different gaming experience from the
original.
6
Graphics Requirements Immersing the Player

• Good game visuals are important to both drawing a
  player into a game and keeping them there.
• The player should be continuously bathed in a
  stream of attractive and fitting graphical
  effects.
• Visual effects are not the only way to immerse a
  player, but poor visuals can prevent a player
  from being drawn in.

7
Graphics Requirements Immersing the Player
Screen shot from Rune. The stunning visuals in
this game go a long way towards immersing the
player. You can easily lose hours in a
gameworld like this!
8
Graphics Requirements Support Storytelling

• Graphics contribute to storytelling aspects of a
  video game in a variety of ways.
• Introducing and developing characters, and
  conveying their physical, emotional, and mental
  state.
• Establishing the setting of the various levels
  and locales within a game.
• Establishing the general atmosphere, tone, and
  mood of a game.

9
Graphics Requirements Support Storytelling
Screen shot from the Legend of Zelda The
Ocarina of Time. Visuals such asthis one from
the introduction establish setting and a
foreboding mood. Therepresentation of Ganondorf
clearly casts him as the games villain.
10
Graphics RequirementsProviding Feedback

• The graphics used in a game should provide good
  feedback to the user.
• Every user action should result in some kind of
  visual feedback.
• This feedback should provide useful information
  to a player and indicate whether their actions
  were successful or not. (If not, they should
  also suggest corrections.)
• Without good visual feedback, it is easier for
  players to get lost or frustrated.

11
Graphics RequirementsProviding Feedback
Screen shot from Oni. Oni provides good visual
feedback to the user. Everyaction has a visible
result. Useful information is also conveyed for
example,the colour of an attacks impact
indicates an enemys remaining health.
12
Graphics RequirementsBe Consistent

• To support immersion, it is important to be
  consistent in game visuals.
• Nothing jolts you out of immersion quicker than
  something that is visually out of place.
• If something is represented in a certain way,
  make sure it is always done this way.
• Use the same models, same textures, same bitmaps,
  same colour schemes, and so on.
• If something is suddenly done differently, a good
  reason should be provided. Otherwise, it appears
  inconsistent.

13
Graphics RequirementsBe Consistent
Screen shot from Die Hard Nakatomi Plaza. The
graphics are decent, butthere is an
extraordinary lack of variety in opponents. You
shoot someone,round the corner, and there he is
again such consistency is inconsistent!
14
Graphics RequirementsProvide a Good Viewpoint

• It is critically important to visually present
  the player with a view of the game that makes the
  game easier to play.
• Viewpoints that make the action difficult to
  follow make a game frustrating and hard to play.
• Whenever possible, allow the user to have control
  over how they will view the game.
• If you find, however, that they constantly need
  to adjust their viewpoint, this is an indication
  of potential trouble.

15
Graphics RequirementsProvide a Good Viewpoint
Screen shot from Splinter Cell. A very
impressive game, but some people find they spend
as much time moving the viewpoint as moving their
character.
16
Graphics RequirementsAllow Tuning

• It is important to be as flexible as possible
  with the graphics used in your game.
• Some users will want to maximize the resolution,
  details, lighting effects, and so to provide the
  most stunning visuals possible.
• Others will lack the hardware to do so, or prefer
  a smoother and quicker game over one that is
  better looking, but slower.
• The more you allow the user to tune the visuals
  provided by the game, the better.

17
Graphics RequirementsAllow Tuning
Screen shot from Unreal Tournament 2003. There
are a lot of graphics features that can be tuned
by the user, both interms of basic display
options, and details as well.
18
Graphics RequirementsMake an Impact

• Basically, you want the player to say Wow! That
  looks amazing!
• You want to catch the players attention, and get
  them interested in the game to see more.
• Once they are hooked, your games immersion will
  keep them that way.
• You want your graphics to be stunning and
  memorable.

19
Graphics RequirementsMake an Impact
Screen shot from Halo. The stunning visuals in
this game can make ahuge impact. With scenes
like this, you just want to sit back anddrink it
all in.
20
Graphics RequirementsOne Last Note

• It is important to point out that things such as
  realism are not part of the requirements of
  graphics for video games.
• In some games and game genres, this can be very
  important.
• Sometimes, however, other styles of visual
  representation are more fitting and result in a
  better overall game.

21
Graphics RequirementsOne Last Note
Screen shot from Jet Set Radio Future. With its
cartoon-style cel-shading, there is a very
definite look and feel to the game. It would be
hard toimagine the game any differently.
22
Graphics RequirementsOne Last Note
Movie from Robotech Battlecry. The cel-shading
using in this gametakes one back to many
Saturday mornings well wasted with thecartoon
series. Difficult to picture it any other way.
23
Graphics Techniques

• Producing graphics for video games is typically
  composed of three stages
• Modeling
• Rendering
• Display
• Each stage is important to producing good visual
  effects for a game.

24
Modeling Techniques

• Modeling for graphics in video games involves
  providing some kind of description of the various
  objects, characters, and parts of levels in a
  game.
• These descriptions define the general look and
  feel for a game.
• These descriptions tend to be formal or otherwise
  mathematically rigorous.
• By providing such a description, rendering can
  more easily occur in the next stage.

25
Modeling Techniques

• Depending on the kind of game being developed,
  the target platform, and resources available,
  there are several general techniques.
• Two dimensional modeling.
• Three dimensional modeling.
• Fractal modeling.
• Some combination of the above.

26
Two Dimensional Modeling

• Most early games used graphics that were totally
  two dimensional.
• This was the easiest and cheapest method given
  the technology at the time.
• Most modern games use three dimensional modeling,
  but some still use two dimensional modeling
  instead.
• All of the past two generations of consoles had
  three dimensional support in hardware.
• Most home computers now have this hardware
  acceleration technology as well.

27
Two Dimensional Modeling
Screen shots from Pac-Man (left) and Donkey Kong
(right). Both made use oftwo dimensional
graphics, like most early video games.
28
Two Dimensional Modeling

• Graphics in two dimensional games are composed of
  bitmaps.
• Each bitmap is essentially a two dimensional
  array of pixels that, when rendered to the
  screen, produce an image.
• Each character, object, and piece of a level is
  composed of such a bitmap.
• Each bitmap has an associated palette or colour
  map that lists the colours used by pixels in the
  image.
• Palettes can be either shared or private.

29
Two Dimensional Modeling

• The term sprite is used to refer to a bitmapped
  object with transparency areas.
• Transparency is typically provided either by
  using a mask or by designating a particular
  bitmap colour to be transparent.
• This allows bitmaps to be non-rectangular and
  lets the background show through.
• Typically, game characters and the objects they
  interact with are sprites.

30
Two Dimensional Modeling
Bitmap of Mario from Donkey Kong. Mario was
composed of pixels of three colours. Mario was
also a sprite in the game the black area
aroundMario in the above picture was transparent
in the game.
31
Two Dimensional Modeling

• Two dimensional modeling has its pluses
• Relatively cheap and easy to use.
• Good for beginning developers.
• Collision detection is relatively simple a
  matter of detecting overlapping bitmaps.
• Animation is also fairly simple cycle through
  multiple bitmaps for the same object.
• It also has its drawbacks
• Realism and certain effects can be hard to
  achieve.
• With the advent of three dimensional modeling,
  many people believe two dimensions is archaic.

32
Two Dimensional Modeling
Screen shot from Metal Slug. Even though it, and
the sequels in theseries have been
two-dimensional graphically, all are still quite
popular.
33
Three Dimensional Modeling

• Three dimensional modeling in video games is
  quickly growing in popularity.
• Hardware support in consoles and computers is now
  quite common.
• Features and performance allow for better visuals
  in games at little cost.
• Software support for developers is improving,
  making this approach easier.
• This form of modeling, however, has been around
  in some form since 1980.

34
Three Dimensional Modeling
Screen shots from Battlezone (left) and I, Robot
(right). In 1980, Battlezonebecame one of the
first three dimensional games, albeit in
wireframe only.In 1983, I, Robot became the
first video game to use three dimensionalfilled
polygons, instead of just a wireframe mesh.
35
Three Dimensional Modeling

• There are many three dimensional modeling
  techniques used in the video game industry.
• These include
• Polygonal modeling.
• Bicubic parametric patches.
• Constructive solid geometry.
• Spatial subdivision techniques.
• Implicit representation.

36
Polygonal Modeling

• This is currently the most popular approach to
  modeling in games.
• In this approach, objects are represented by a
  mesh of planar polygons.
• The accuracy of the representation is limited
  only by the number of polygons used in
  constructing the mesh.
• Continuity between polygons can be achieved
  through proper shading of the polygons. This
  provides a smooth surface with no visible breaks
  or lines.

37
Polygonal Modeling
Polygonal model from Oni. This model shows a
wireframe representation of the main character
Konoko.
38
Polygonal Modeling

• Achieving very high accuracy with polygons alone
  can be quite costly.
• Creating models with high polygon counts can be
  time consuming and expensive.
• Pushing the high number of polygons out to video
  hardware can be resource intensive.
• To increase accuracy without increasing the
  polygon count, textures are mapped on to the
  polygons in models.
• When this applies to characters, this process is
  sometimes referred to as skinning.

39
Polygonal Modeling
Model from Heretic II. On the left is a
wireframe version of the model on the right is
the skinned version.
40
Polygonal Modeling

• Polygonal modeling has several benefits
• Realistic models are reasonably simple to
  construct.
• Hardware acceleration for this approach is good.
• Software tool support is also quite good.
• It also has some drawbacks
• Non trivial for beginner developers.
• Collision detection and animation become more
  complex and more difficult.
• Frequently a lot of waste. When viewing objects
  from a distance, you can be using thousands of
  polygons to model a handful of pixels on the
  screen.

41
Bicubic Parametric Patches

• The representation in this case is similar to a
  polygon mesh, except that the individual polygons
  are now curved surfaces called patches.
• Each patch is specified by a mathematical formula
  giving its position and shape.
• The shape or curvature of each patch can be
  changed by tuning the mathematical specification.
  This allows for interesting interactive
  possibilities.
• There are several approaches to doing this,
  including Bezier curves, B-spline curves, and
  many more.

42
Bicubic Parametric Patches
The Utah Teapot represented by bicubic parametric
patches. Part a)depicts a single shaded patch.
Part b) depicts the patch edges. Part c)depicts
a wireframe of the control points of the mesh.
43
Bicubic Parametric Patches

• Bicubic patches can require fewer resources to
  provide an equivalent representation to a
  polygonal mesh.
• The Utah Teapot on the previous slide requires
  only 32 patches a comparable polygonal mesh
  would require roughly 2048 polygons instead.
• Bicubic parametric patches have problems,
  however.
• Some problems in maintaining smoothness.
• More complex and more difficult to work with.
• Less hardware and software support.

44
Constructive Solid Geometry

• This is an exact representation of objects to
  within certain rigid shape limits.
• Many manufactured objects can be represented by
  some combination of elementary shapes or
  geometric primitives.
• Constructive solid geometry usually works with
  the following
• Primitive volumes spheres, cones, cylinders,
  prisms (rectangular and triangular), and so on.
• Basic operators Boolean set operators (union,
  intersection, difference, and so on), and a
  variety of linear transformations.

45
Constructive Solid Geometry
Example of constructive solidgeometry. By using
union, differenceand intersection operators
inparts a), b), and c) respectively, complex
objects can be modeledaccurately and precisely.
46
Constructive Solid Geometry

• Advantages of constructive solid geometry
• Good for describing manufactured objects.
• Works best with a rich set of primitives.
• Moderately intuitive and easy to understand.
• Disadvantages of constructive solid geometry
• Some objects may be difficult to describe.
• We are limited by the set of primitives and
  operators that apply to them.

47
Spatial Subdivision Techniques

• In this approach, the game world object space is
  divided into elementary cubes, known as voxels,
  and each voxel is labeled as empty or containing
  part of an object.
• This is the three dimensional analog of
  representing a two dimensional object as a
  collection of pixels in a bitmap.
• In three dimensions, we are representing the
  space occupied by an object, not just its
  surface.
• There are many different ways of carrying out
  this division.

48
Spatial Subdivision Techniques Octrees

• An octree is a data structure formed by
  repeatedly dividing a cubic region of the object
  space into 8 smaller subcubes.
• Each subcube is a child node of the parent cube
  node in the tree data structure.
• This process is repeated until the subcubes are
  of an appropriate size to accurately represent
  the desired object.

49
Spatial Subdivision Techniques Octrees
An octree showing the parent cube, and its 8
subcubes.
50
Spatial Subdivision Techniques BSP Trees

• A Binary Space Partitioning Tree, or BSP tree,
  partitions the game world object space into two
  parts by using a splitting plane.
• Each partition results in two children nodes in a
  BSP tree under the parent space node.
• A BSP tree can be used to represent an octree,
  but it is much more flexible.
• Partitions do not have to divide the space
  equally, and they only need to divide the space
  where needed. They can also divide space at
  arbitrary orientations.

51
Spatial Subdivision Techniques BSP Trees
Screen shot from Quake II. Many games, including
the Quake series use BSP trees for representing
elements of game levels.
52
Implicit Representation

• Implicit functions are sometimes used as a method
  of object representation.
• For example, an implicit function for defining a
  sphere is x2 y2 z2 r2
• This method, however, is not frequently used in
  gaming.
• It is difficult to represent all but a few
  objects using this method.
• It is also a very inconvenient form when it comes
  to do rendering.

53
Fractal Modeling

• Many natural objects cannot be represented easily
  using traditional methods.
• Natural objects tend to be irregular, have
  fragmented features, or are self-similar.
• For example clouds, terrain, mountains, water,
  plant life, and so on.
• Such objects can typically be described well
  using fractal geometry.

54
Fractal Modeling
Screen shot from Scorched3D. It uses fractal
geometry to generate its terrain, water, clouds,
and fog effects in a natural way.
Becauseterrain is generated and not stored, you
get a different game each time.
55
Fractal Modeling

• In fractal geometry, procedures are used to
  describe how to generate the object, as opposed
  to defining what the object is.
• Fractal objects have two characteristics
• Infinite detail at every point.
• Self similarity between parts of the object and
  the overall features of the object.
• How these properties exhibit themselves depend on
  the fractal representation.

56
Fractal Modeling

• A common way of generating simple self similar
  fractals involves the use of an initiator and a
  generator.
• Parts of the initiator are repetitively replaced
  by the generator over and over.
• For example

57
Combinations of Models

• Some games use a combination of two dimensional,
  three dimensional, and fractal modeling
  techniques.
• For example
• Some two or three dimensional games use fractals
  to model natural objects and phenomena in the
  game world.
• Some two dimensional games use three dimensional
  models that are prerendered into bitmaps used to
  represent sprites or background objects in the
  game.

58
Combinations of Models
Screen shot from Diablo II. The game engine is
two dimensional, but mostgraphics came from
prerendered three dimensional models.
59
Rendering Techniques

• Rendering is the process of taking a model of the
  game world and preparing it for display on the
  screen.
• This includes many things, including
• Composing the scene.
• Clipping the world to the viewable part.
• Adding shading, lighting effects, and other
  special effects.
• The tasks required often depend upon the type of
  modeling used.

60
Two Dimensional Rendering

• When one thinks of rendering, one rarely thinks
  of two dimensional graphics.
• There are, however, several tasks required to go
  from game element models to the screen for
  display to the player.
• The scene to display must be composed. Sprites
  must be layered in position on the background
  according to their relative priorities.
• Since it would be rare for the entire game world
  to be displayed at once, it must be clipped so
  the relevant parts are in focus.

61
Two Dimensional Rendering
Screen shot from Double Dragon. Sprites
representing the various combatantsmust be place
onto the screen in front of the background. They
are alsoprioritized, according to their depth on
the screen. Further, since the game only shows
part of a level at a time, the game world must be
clipped to the screen.
62
Three Dimensional Rendering

• Three dimensional rendering is typically more
  complex than two dimensional rendering.
• This complexity provides additional realism and
  other benefits.
• There are typically two kinds of rendering for
  three dimensional models
• Prerendering (for example, ray tracing)
• Real-time rendering

63
Prerendering

• The philosophy behind prerendering is to carry
  out computationally intensive rendering tasks
  offline in advance.
• To provide three dimensional graphics to a two
  dimensional game.
• To reduce the hardware requirements of a game.
• To improve the level of details provided by game
  graphics.
• There are many methods to perform this
  prerendering for a game.
• The most common is ray tracing.

64
Prerendering Ray Tracing

• Ray tracing is a rendering technique that takes
  advantage of the way our eyes see to generate a
  stunning image.
• We see by light emanating from a source bouncing
  off of objects in the world and into our eyes.
• Ray tracing replicates this process backwards,
  starting from our eye and working back towards
  the light sources.
• Hence, ray tracing is more properly referred to
  as backwards ray tracing.

65
Prerendering Ray Tracing
Light Source
Internally Reflected Ray
Light Rays
Reflected Ray
Refracted Ray
Pixel Ray
66
Prerendering Ray Tracing
Three of these images were produced usingray
tracing, and one was not.Can you tell which one?
67
Real-time Rendering

• Methods such as ray tracing are too
  computationally expensive to carry out in real
  time.
• Other methods are needed. Typically, you need to
  do the following tasks
• Composing the scene.
• Clipping to the view volume.
• Culling and hidden surface removal.
• Shading, texturing, lighting, and other effects.

68
Real-time RenderingScene Composition

• Scene composition involves placing game objects
  into the game world for later rendering.
• This typically involves the following
• Determining the background.
• Determining object positions.
• Determining object orientations.
• Determining object relationships.
• Determining lighting conditions.

69
Real-time RenderingView Volume Clipping

• View volume clipping determines the portion of
  game world being viewed and the set of objects to
  be rendered.

View Volume
View Frustum
Viewing Direction
View Plane
Near ClipPlane
Far Clip Plane
70
Real-time RenderingSurface Removal

• Surface removal eliminates entire objects, entire
  polygons, or partial polygons from the rendering
  process.
• Culling or back-face elimination compares the
  orientation of polygons with the direction of
  viewing and removes those polygons that cannot be
  seen.
• Hidden surface removal deals with the situation
  when polygons partially obscure others using some
  form of depth sorting.

71
Real-time RenderingSurface Removal
Original Scene
After Culling
After Culling andHidden Surface Removal
72
Real-time RenderingShading

• Shading involves filling in polygons with the
  appropriate colours. Shading also tries to
  smooth polygonal seams as necessary to provide
  continuity.
• There are many kinds of shading
• Flat shading shading with a single intensity
• Gouraud shading shading with interpolation to
  smooth polygonal edges
• Phong shading shading with interpolation and
  illumination applied
• Cel shading shading to provide a cartoonish look

73
Real-time RenderingShading
Screen shots from Virtua Fighter (left) and
Virtua Fighter 4 (right). Theoriginal
primarily used flat shading with some lighting
effects very little wasdone to smooth seams
between polygons. Virtua Fighter 4 uses much
more advanced techniques for a highly polished
look.
74
Real-time RenderingShading
Screen shot from Jet Set Radio Future. As
opposed to traditional shading models for
providing realistic graphics, cel shading is an
increasingly popularapproach for a cartoonish
look and feel.
75
Real-time RenderingTexture Mapping

• Instead of filling polygons using some kind of
  shading method, polygons can be filled with a
  texture to add details without increasing polygon
  count.
• Textures tend to be either two dimensional
  bitmapped images, or some kind of function
  describing how to generate the texture.
• There are many different texturing techniques,
  each with their own advantages and disadvantages.

76
Real-time RenderingTexture Mapping

• The images to the right show the difference
  between shading (top) and texture mapping
  (bottom).
• Texturing has added great detail without
  requiring any additional polygons.

77
Real-time RenderingBump Mapping

• Although texture mapping can be used to add
  surface detail, it cannot capture surface
  roughness adequately.
• The illumination detail in the texture typically
  does not match the scene illumination.
• Bump mapping applies a perturbation function or
  map to the surface normal of a surface and uses
  the perturbed normal in illumination
  calculations.
• This method is simple yet impressive.

78
Real-time RenderingBump Mapping
Bump mapping can produce some really interesting
effects, as shown in this screen shot and demo.
79
Real-time RenderingEnvironment Mapping

• Environment mapping, also known as reflection
  mapping, is a shortcut to rendering shiny objects
  that reflect the environment in which they are
  placed.
• The goal is to provide some of the same realistic
  effects as ray tracing in real time, without the
  computational expense.
• Special environment textures are prerendered and
  then mapped on to reflective surfaces in
  real-time as needed.

80
Real-time RenderingEnvironment Mapping
By enclosing objects in a sphere or cube of
environment maps, interesting effects can be
achieved as shown in this screen shot and demo.
81
Real-time RenderingLighting and Shadows

• There are many lighting techniques that can be
  used in rendering a game scene.
• Static lighting lights that do not move their
  effects can be precomputed and stored as a two
  dimensional lighting map.
• Dynamic lighting lights that move or change
  their effects must be calculated on the fly.
  Dynamic lights can also be switched on and off,
  or possibly even destroyed.
• Shadowing can therefore also be static or dynamic
  in nature.

82
Real-time RenderingLighting and Shadows
Screen shot from Splinter Cell. Splinter Cell is
widely considered to havesome of the best
lighting and shadow effects in a video game to
date.
83
Real-time RenderingLighting and Shadows

• There are also other lighting effects that show
  up in video games
• Lens flares they occur in photography when a
  bright light passes in front of a camera,
  creating various glares on each of its lenses.
  They can actually be done simply in two
  dimensions without knowing the actual physics
  behinds them.
• Particle effects various effects for things such
  as explosions, smoke, fog, and so on.

84
Real-time RenderingLighting and Shadows
A sample lens flare. It is interesting that game
developers strive to createrealistic lens flare
effects, while at the same time, photographers
doeverything they can to avoid them!
85
Real-time RenderingLighting and Shadows
Particle effects can add greatly to the lighting
in a game, as shown in this image and demo.
86
Display Methods

• Once game models have been rendered, they are
  ready for display.
• There are a variety of application programming
  interfaces (APIs) and software development kits
  (SDKs) suitable for displaying graphics.
• The choice usually depends on the type of
  modeling used.
• Some methods are geared towards two dimensional
  graphics, while others are geared towards three
  dimensional graphics.

87
Display MethodsTwo Dimensional Methods

• DirectX
• Microsofts SDK for its Windows platforms.
• Simple DirectMedia Layer (SDL)
• A cross platform SDK capable of both two
  dimensional and three dimensional display.
• Java 2D
• For the cross platform Java Virtual Machine.
• X Windows
• A Unix display system. Several graphics toolkits
  build on top of it provide better features Xt,
  Athena, Motif, Forms, QT, GTK, and so on.

88
Display MethodsThree Dimensional Methods

• Glide
• A proprietary API developed by 3DFX no longer
  widely in use.
• Direct3D
• Microsofts SDK for its Windows platforms.
• OpenGL
• An open standard available for multiple
  platforms.
• Simple DirectMedia Layer (SDL)
• A cross platform SDK capable of both two
  dimensional and three dimensional display.
• Java 3D
• For the cross platform Java Virtual Machine.