Backing up the planet digitising culture history and heritage
[Music]
hello
so what is photogrammetry
photogrammetry was conceived in 1851 the
same era as photography itself
it was used for the purpose of map
making and surveying but wouldn’t really
come through
until the late 20th century
so the process of photogrammetry is
pretty simple yet it’s like magic
you can take a bunch of photographs of
an object or an environment
and produce extremely dense 3d point
clouds from this
how it does it is it distinguishes
similar features between
similar photographs so with the event
of graphics cards recently in the last
five ten years we’ve seen a rapid rise
in this
the gpu acceleration that we’ve had
offered is unbelievable
so we were where we were struggling with
this uh 10 years ago
we could only do a few million points of
detail and it’ll take months of
processing time
with a few hundred photographs now we
can do tens of billions
in days with thousands of photographs
and so the big problem is
how do you manage all this data because
the problem is simple
traditional means take minutes if not
hours to render a single frame
and we need to see this in real time
so we developed
an ultra-realistic sorry a real-time
ultra-realistic visualization
methodology
to get all this amazing point cloud data
of these amazing environments
live so you can feel it so you can see
it
and so we caught single single-pass
photogrammetry
photogrammetry for lazy people
the benefits of this are pretty clear
you get immediate response immediate
gratification of these scenes
um and it’s the level of detail we’re
able to retain as a result of this
you can get right up see every paint
stroke every nook and cranny
and in real time with real time lighting
the whole lot
it benefits everything and
once again level of detail we haven’t
lost a thing
we do not lose a single thing in this
process we retain
all those tens and hundreds of billions
of points and we get it running on real
time on a standard desktop pc
or mobile web browser
so what makes us different from the
competition is the perfection of
imperfection shimmer and shine
the fact that the way wood reflects on a
wooden varnish floor or the way that
carpet is with
you know rocks or whatnot it’s it’s
these levels of lighting imperfections
that really create
the sense of immersion which then
creates a sense of retention
um so yeah r d is really a big thing
here
we’re constantly trying to make
ourselves obsolete so locations we have
done
uh nifitari queen nephite and the valley
of the queens tooth carmen
king tut that was fun uh carmack
new local sites like uh oh sorry soon
large hydrogen collider
we’ve done amazing sites in new zealand
as well uh new zealand parliament
the treaty of waitangi which was quite
frankly an honour
uh the homestead sir james wallace
beautiful beautiful place that you saw
before
i was literally the epitome of living
the urban lifestyle
extremely house proud had had this home
for eight and a half years
renovated it myself went on youtube lent
i’d do it all sand the floors a whole
lot
proud father to a beautiful daughter and
my loving partner chantal
it was the morning after my 31st
birthday party
uh we woke up to flames coming through
howling through our skylight one day
uh and we knew exactly what had happened
we lost the home the whole place was
gutted
and so i just got this crazy obsession
how do i get back to the house how did
me and my daughter return home
so 2008 i discovered this i remember
watching this amazing talk
ted talk photosynth and this was the
precursor of photogrammetry
and i knew i’d had eight and a half
years of family photographs
and so i was like well why don’t i just
use all my family photographs to
recreate my house let’s see if this
is doable so come 2015 when this
technology was really maturing
it was like magic absolute magic
so i wasn’t classically trained apart
from my commodore amiga in 1989.
thank you mum honestly changed my life
i’m an amiga boy
and so that’s the only experience i’d
actually had as a kid
playing with 3d packages i had no idea
what i was doing
so i didn’t know what rules i was
breaking all i knew is i needed to get
from a
to b i needed to get from here to here
didn’t bother
did not want to be stuck in this
traditional 20 year old
20 year old vfx pipeline and so my
process was unorthodox
first scene we did alleyway
uh 2015 this just took the internet by
song
i look at it now i cringe but hey but
you know it’s really funny alleyways
and shoes for us photogrammetrists seem
to be really a rite of passage i have no
idea why
we always always seem to gravitate
towards this
so why is importance of encapsulating
culture so important
we have environmental degradation on a
scale we’ve never seen before due to
global warming climate change
we’re seeing the sixth holistine
extinction
you know this is since the planet begun
guys
acts of god fire floods
political unrest war these are all
factors we are losing culture at a rapid
rate
uh ironically i was asked i was
propositioned by the bbc two years ago
to um go and go through some isis
tunnels
because apparently the isis terrorists
war fighters i don’t know what to call
them had discovered the syrian treasures
so syria turkey brazil lost the national
museum
up in flames a year ago fire is a fema
and the reason why we need to provide
these experiences everyone is it’s
really simple
it goes along the lines of world peace
basically because
good luck finding a well-traveled bigot
as i like to say
to live john to be living john malkovich
to be living in someone else’s shoes it
gives you a sense of culture and
identity it makes you realize how
similar we all really are
and i had my own biases being in new
zealand for 35 years of my life i’d
never traveled
right and so i always thought you know
when i was asked to go to iraq i always
thought that arabic was going to be
harsh and angry sounding because of the
media and the way i was portrayed
it was phenomenal fathers and fathers
and mothers
just raising and loving their sons and
daughters everyone just living day by
day
blew my mind and then look we need to
make accessibility for everyone
and this is very close to heart the
disabled the elderly
you know hospice care people who might
not see the last great wonders of the
world
you know we can give this to them now
but it’s also inaccessibility many of
these sites you just can’t get to
anymore
or they require very skilled teams like
ourselves to get in there
and give it to the rest of you but as it
relates to me i’m legally blind by
definition of the new zealand government
five percent one eye 2.5 another and it
does make me laugh
that they let me into these places i
remember dealing with like
king touched jandles and i’m like please
guys don’t move a thing well five
security guard type i don’t know what
they call them like uh security military
police
are holding ak-47s around me i hadn’t
even seen a single gun in my life at
this point
and uh yes it was kind of a an
experience but also the visual aids that
this provides me personally so when i
was doing queen nefertari
and in this location i could never
actually appreciate the hieroglyphs
you know i can’t see a meter in front of
my face and so when i actually got it
into the vr
i was able to get up close and personal
genuinely genuinely intimate
and that is it is a weird thing to be
able to have a better experience
in vr than it is in the real life for
myself
we like to think of visas teases to the
movie you know
we find that these experiences actually
encourage tourism
they don’t digress from it and also the
fact is many of these sites are going to
be
gone or locked up soon and people won’t
be able to see them notre dame is a fine
example of us
we will provide the mains for them to
experience this over the next five years
and so the history books really are
the history books of the future is what
we’re basically in the business of
i love the fact that my daughter is
going to be able to enjoy these
experiences and social studies
in years to come and i love the fact
that it makes me very proud as a father
knowing that many of these experiences
will be us
now the problem is who owns history
there’s a big issue of bureaucracy of
academia right now
i’ve traveled the world and i’ve talked
with many universities
and look i’m not i do not have a
master’s
i have a bachelor of recording arts i’m
an audio engineer guys
but there is a there’s a bit of a bias
going on it’s a bit close
and a lot of waste and a lot of time
there’s no sense of urgency
in these in these entities but the
bigger culprits guys
really they’re the corporations
they’re doing what is could be only said
as a digital land grab right now
and it’s quite disturbing they’re
hoarding massive amounts of data
and the fake pretense saying that they
are going to provide the star to the
public to for
people like myself to do things with it
they’re providing us low resolution
jpegs
when we need the high resolution raw
photography you can google them
and so we believe we are on the right
side of history in this
uh my unique situation allowed us to
have
a fundamentally stronger way of us
because it was a passion project
so machine learning to the rescue
mundane tasks take us so with queen
nefertari for an example
any one of our experiences it takes
three guys six weeks
and about twenty thousand dollars to
produce one of these experiences
which is actually considerably cheaper
than any of the competition and
considerably faster
but we can’t do it at this pace we need
to find better ways
you know i refer to the tasks as pushing
pixels
it’s exhausting but luckily we’re an r d
company and so when deep learning
machine learning came to be a thing
we already had all the computing power
we already had
all the resources we’ve required but
most importantly we had plenty of data
we had tens of you know hundreds of
thousands of photographs that we could
use to train the system
and so why not teach the computer to do
these boring tasks
one of these tasks is uh what we call
contextually intelligent interpretation
and painting basically we’re able to go
and lesser around something we don’t
want in one of these scenes
once again with never tire we had wooden
floorboards we had no smoking signs we
had plaques we had halogen lighting
circle around and the computer knows how
to fill it in with something that is
contextually artistically correct
uh other reasons why this is so useful
is quite frankly because
where we get occlusion issues in our
photogrammetry we don’t take enough
photographs
this is able to look at those low
density areas and fill it in
artistically as good as any human could
do
if not better semantic image
segmentation this is the ability for us
to essentially
optimize and divide our meshes um divide
our environments to work faster on
machines
and also tag them with basic principles
like wooden floors
have these properties uh carpet has
these properties
and this is where it gets really cool
for audio instead of someone having to
manually go in
and label each one of these objects with
certain properties
we can now have it done automatically
and so this creates
way better immersive experiences for
audio because things reflect sound the
way you’d expect them to reflect
noise removal now this is this is like
magic
there’s your image there’s what we get
that’s your base truth and so we’ve been
able to look why this is useful
is not all photographs are equal right
and so we basically have a situation
where we can enhance
greatly and so how do we use this we can
resurrect
archival footage we can resurrect stuff
that did not
was never intended for photogrammetry
and so
uh i was actually recently asked by
time magazine and my good friend peter
martin were in the running
to do martin luther king’s i had a dream
speech use all that old archival footage
and see if we can bring it back to life
um but christchurch cathedral that’s
something that we really have been
actively looking at so we’re looking at
all the old footage
all the old wedding photos we did a big
call to action about a year ago
trying to archive as much data as we can
to see if we can recreate the church and
it’s in its glory and it’s beauty
uh dppr is our most immediate and this
is like live now we have an early alpha
that
people can try out this technology is
able to look at a photograph
and extrapolate it to its true
fundamentals
uh what that means is we can remove
harsh lighting we can move highlights
um it does a whole bunch other
techniques like super sampling and noise
removal as well
but where it is really good is that with
photogrammetry when we go to these
environments we have baked lighting
depending on where the sun is you know
and you don’t want that if you want to
be able to realize an environment you
need to remove it all
and this is a massive deal for the
photogrammetry community but we also
were able to create byproducts of us so
we can a single person can now take a
photo
and essentially produce all the texture
maps they need for any vfx package
or game engine of their choice and this
is
incredible so an individual one guy can
actually do more now
for what a team of 30 could do in three
months
so the need for economies of scale we
have an arsenal of tools
we have mass automation we
this allows us to have economies of
scale and reduce costs dramatically
so we can do more now i can’t be
everywhere
okay as much as i like flying i actually
hate flying but as much as i like
traveling
um we need people on the ground to
actually
acquisition the starter but we need to
create incentives to allow them to want
to do this
and so we’re saying let’s introduce an
artist’s rights management system
where that photograph is
you know is basically secured by a
digital ledger so where that photograph
that individual is tagged
they can say what that photograph is
used for and what use cases
um so it’s think of it like creative
commons in a different level of creative
commons
so we need to make this uh
basically work for everyone and not
cross any lines basically
and also the artist needs to get paid
royalties
this is important so for now
philanthropic entities we believe in the
short term can jump in and fund many of
these endeavors very similar how we did
to antarctica
these great expeditions of the world of
past centuries gone
and these parties were you know to
mention in the history books we know we
know who the heroes were who went to
these
places and we know the people who
bankrolled it there’s nothing wrong with
that
licensing to institutes universities
schools educational facilities the
museology scene museums
they are dying they’re craving for these
kind of experiences and right now
they’re
having to spend hundreds of thousands of
doing it for themselves
and it’s usually closed off we don’t
want that we want everyone to be able to
experience this
now the exciting thing that um i spend a
lot of time in la
uh what film studios are doing now with
real-time virtualization
we’re they’re shooting everything on
green screen now and they’re going out
and getting these photogrammetry
environments and placing people on them
we’re saying look instead of you sending
a team of 20 to do it
license that environment through us and
we’ll provide you a fraction of the cost
so we have a decentralized backup
peer-to-peer none of these photos stay
in one place it’s encrypted
but it’s distributed and we have a
marketplace that benefits everyone
so the conclusion guys an army of
millions
running around taking these photographs
acquisitioning this data
a decentralized secure database that
protects the starter
computing power and ai deep learning
making this an
automated process
and we as a result get
mass real-world encapsulations for
everyone
and this is how we’re going to back up
the planet