Animating a photoreal digital face Paul Debevec
one of the biggest challenges and
computer graphics has been able to
create a photo real digital human face
and one of the reasons that’s so
difficult is that unlike aliens and
dinosaurs we look at human faces
everyday they’re very important to how
we communicate with each other and as a
result we’re tuned into the subtlest
things that could possibly be wrong with
the computer rendering in order to
believe whether these things are
realistic and what I’m going to do in
the next five minutes is take you
through a process where we tried to
create a reasonably photorealistic
computer-generated face using some
computer graphics technology we’ve
developed and also some collaborators at
a company called image metrics we’re
gonna try to do a photo real face of an
actress named Emily O’Brien who’s right
there and that’s actually a completely
computer-generated rendering of her face
and by the end of the talk we’re gonna
see it move the way that we did this is
we tried to start with Emily herself who
was gracious enough to come to our
laboratory in Marina del Rey and sit for
a session in Light Stage five this is a
face scanning sphere with 156 white LEDs
all around that allow us to photograph
her in a series of very controlled
illumination conditions and the lighting
that we use these days looks something
like this we shoot all of these
photographs in about three seconds
and we basically capture enough
information with video projector
patterns that drape over the contours of
her face in different principal
directions of light from the light stage
to figure out both the course scale and
the fine scale detail of her face if we
zoom in on this photograph right here
you can see it’s a really nice
photograph to have a fur because she’s
lit from absolutely everywhere at the
same time to get a nice image of her
facial texture and in addition we’ve
actually used polarisers on all the
lights just like polarized sunglasses
can block the glare off of the road
polarisers can block the shine off of
the skin so we don’t get all those
specular reflections to take this map
now if we turn the polarisers around
just a little bit we can actually bring
that specular reflection of the skin
back in and you can see she looks kind
of shiny and oily at this point and if
you take the difference between these
two images here you can get an image lit
from the entire sphere of light of just
the shine off of Emily’s skin I don’t
think any photograph like this had ever
been taken before we’ve done this and
this is very important light to capture
because this is the light that reflects
off of the first sir
the skin doesn’t get underneath the
translucent layers of the skin and blur
out and as a result it’s a very good cue
to the detailed shape of the skin pore
structure and all the fine wrinkles that
all of us have the things that actually
make us look like real humans so if we
use information that comes off of the
specular reflection we can go from a
traditional face scan that might have
the gross contours of the face and the
basic shape and augment it with
information that puts in all of that
skin pore structure and fine wrinkles
and even more importantly since this is
a photometric process that only takes
three seconds to capture we can shoot
Emily in just part of an afternoon in
many different facial poses and
different facial expressions so here you
can see her moving your eyes around
moving her mouth around and these were
actually going to use to create a photo
real digital character so if you take a
look at these scans that we have of
Emily you can see that the human face
does an enormous amount of amazing
things as it goes into different facial
expressions you can see things not only
the base shape changes but all sorts of
different skin buckling and skin
wrinkling occurs you can see that the
skin pore structure changes enormous Lee
from stretched skin pores to the regular
skin texture you can see the furrows in
the brow and how the microstructure
changes there you can see muscles
pulling down it flesh to bring our
eyebrows down our muscles bulging in her
forehead when she winces like that in
addition to this kind of high-resolution
geometry since it’s all captured with
cameras we get a great texture map to
use for the face and by looking at how
the different color channels of the
illumination the red and the green and
the blue diffused the light differently
we can come up with a way of conveying
the skin and the computer that instead
of looking like plaster mannequin
actually looks like it’s made out of
living human flesh
and this is what we used to give to the
company image metrics to create a rigged
digital version of Emily we’re just
seeing the coarse scale geometry here
but they basically created a digital
puppet of her where you can pull on
these various strings and it actually
moves her face in ways that are
completely consistent with the scans
that we took in addition to the coarse
scale geometry they also use all of that
detail to create a set of what are
called displacement maps that animate as
well these are the displacement maps
here and you can
see those different wrinkles actually
show up as she animates so the next
process was then to animate her we
actually used one of her own
performances to provide the source data
so by analyzing this video with computer
vision techniques they were able to
drive the facial rig with the
computer-generated performance so what
you’re gonna see now after this is a
completely photo real digital face we
can turn the volume up a little bit if
that’s available which metrics is a
marvelous performance driven animation
company we specialize in high quality
facial animation for video games and
films image metrics is a markerless
performance driven animation company we
specialize in high quality facial
animation for video games and films so
if we break that down into layers here’s
that diffuse component we saw in the
first slide here’s the specular
component animating you can see all the
wrinkles happening there and there’s the
underlying wireframe mesh and that’s
Emily herself now where are we going
with this here we’ve actually gone and
gone a little bit before beyond light
Stage five this is light stage six and
we’re looking at taking this technology
and applying it to whole human bodies
this is Bruce LOM and one of our
researchers in the group who graciously
agreed to get captured running in the
light stage and let’s take a look at a
computer-generated version of Bruce
running in a new environment
and thank you very much