Backing up the planet digitising culture history and heritage

[Music]

hello

so what is photogrammetry

photogrammetry was conceived in 1851 the

same era as photography itself

it was used for the purpose of map

making and surveying but wouldn’t really

come through

until the late 20th century

so the process of photogrammetry is

pretty simple yet it’s like magic

you can take a bunch of photographs of

an object or an environment

and produce extremely dense 3d point

clouds from this

how it does it is it distinguishes

similar features between

similar photographs so with the event

of graphics cards recently in the last

five ten years we’ve seen a rapid rise

in this

the gpu acceleration that we’ve had

offered is unbelievable

so we were where we were struggling with

this uh 10 years ago

we could only do a few million points of

detail and it’ll take months of

processing time

with a few hundred photographs now we

can do tens of billions

in days with thousands of photographs

and so the big problem is

how do you manage all this data because

the problem is simple

traditional means take minutes if not

hours to render a single frame

and we need to see this in real time

so we developed

an ultra-realistic sorry a real-time

ultra-realistic visualization

methodology

to get all this amazing point cloud data

of these amazing environments

live so you can feel it so you can see

it

and so we caught single single-pass

photogrammetry

photogrammetry for lazy people

the benefits of this are pretty clear

you get immediate response immediate

gratification of these scenes

um and it’s the level of detail we’re

able to retain as a result of this

you can get right up see every paint

stroke every nook and cranny

and in real time with real time lighting

the whole lot

it benefits everything and

once again level of detail we haven’t

lost a thing

we do not lose a single thing in this

process we retain

all those tens and hundreds of billions

of points and we get it running on real

time on a standard desktop pc

or mobile web browser

so what makes us different from the

competition is the perfection of

imperfection shimmer and shine

the fact that the way wood reflects on a

wooden varnish floor or the way that

carpet is with

you know rocks or whatnot it’s it’s

these levels of lighting imperfections

that really create

the sense of immersion which then

creates a sense of retention

um so yeah r d is really a big thing

here

we’re constantly trying to make

ourselves obsolete so locations we have

done

uh nifitari queen nephite and the valley

of the queens tooth carmen

king tut that was fun uh carmack

new local sites like uh oh sorry soon

large hydrogen collider

we’ve done amazing sites in new zealand

as well uh new zealand parliament

the treaty of waitangi which was quite

frankly an honour

uh the homestead sir james wallace

beautiful beautiful place that you saw

before

i was literally the epitome of living

the urban lifestyle

extremely house proud had had this home

for eight and a half years

renovated it myself went on youtube lent

i’d do it all sand the floors a whole

lot

proud father to a beautiful daughter and

my loving partner chantal

it was the morning after my 31st

birthday party

uh we woke up to flames coming through

howling through our skylight one day

uh and we knew exactly what had happened

we lost the home the whole place was

gutted

and so i just got this crazy obsession

how do i get back to the house how did

me and my daughter return home

so 2008 i discovered this i remember

watching this amazing talk

ted talk photosynth and this was the

precursor of photogrammetry

and i knew i’d had eight and a half

years of family photographs

and so i was like well why don’t i just

use all my family photographs to

recreate my house let’s see if this

is doable so come 2015 when this

technology was really maturing

it was like magic absolute magic

so i wasn’t classically trained apart

from my commodore amiga in 1989.

thank you mum honestly changed my life

i’m an amiga boy

and so that’s the only experience i’d

actually had as a kid

playing with 3d packages i had no idea

what i was doing

so i didn’t know what rules i was

breaking all i knew is i needed to get

from a

to b i needed to get from here to here

didn’t bother

did not want to be stuck in this

traditional 20 year old

20 year old vfx pipeline and so my

process was unorthodox

first scene we did alleyway

uh 2015 this just took the internet by

song

i look at it now i cringe but hey but

you know it’s really funny alleyways

and shoes for us photogrammetrists seem

to be really a rite of passage i have no

idea why

we always always seem to gravitate

towards this

so why is importance of encapsulating

culture so important

we have environmental degradation on a

scale we’ve never seen before due to

global warming climate change

we’re seeing the sixth holistine

extinction

you know this is since the planet begun

guys

acts of god fire floods

political unrest war these are all

factors we are losing culture at a rapid

rate

uh ironically i was asked i was

propositioned by the bbc two years ago

to um go and go through some isis

tunnels

because apparently the isis terrorists

war fighters i don’t know what to call

them had discovered the syrian treasures

so syria turkey brazil lost the national

museum

up in flames a year ago fire is a fema

and the reason why we need to provide

these experiences everyone is it’s

really simple

it goes along the lines of world peace

basically because

good luck finding a well-traveled bigot

as i like to say

to live john to be living john malkovich

to be living in someone else’s shoes it

gives you a sense of culture and

identity it makes you realize how

similar we all really are

and i had my own biases being in new

zealand for 35 years of my life i’d

never traveled

right and so i always thought you know

when i was asked to go to iraq i always

thought that arabic was going to be

harsh and angry sounding because of the

media and the way i was portrayed

it was phenomenal fathers and fathers

and mothers

just raising and loving their sons and

daughters everyone just living day by

day

blew my mind and then look we need to

make accessibility for everyone

and this is very close to heart the

disabled the elderly

you know hospice care people who might

not see the last great wonders of the

world

you know we can give this to them now

but it’s also inaccessibility many of

these sites you just can’t get to

anymore

or they require very skilled teams like

ourselves to get in there

and give it to the rest of you but as it

relates to me i’m legally blind by

definition of the new zealand government

five percent one eye 2.5 another and it

does make me laugh

that they let me into these places i

remember dealing with like

king touched jandles and i’m like please

guys don’t move a thing well five

security guard type i don’t know what

they call them like uh security military

police

are holding ak-47s around me i hadn’t

even seen a single gun in my life at

this point

and uh yes it was kind of a an

experience but also the visual aids that

this provides me personally so when i

was doing queen nefertari

and in this location i could never

actually appreciate the hieroglyphs

you know i can’t see a meter in front of

my face and so when i actually got it

into the vr

i was able to get up close and personal

genuinely genuinely intimate

and that is it is a weird thing to be

able to have a better experience

in vr than it is in the real life for

myself

we like to think of visas teases to the

movie you know

we find that these experiences actually

encourage tourism

they don’t digress from it and also the

fact is many of these sites are going to

be

gone or locked up soon and people won’t

be able to see them notre dame is a fine

example of us

we will provide the mains for them to

experience this over the next five years

and so the history books really are

the history books of the future is what

we’re basically in the business of

i love the fact that my daughter is

going to be able to enjoy these

experiences and social studies

in years to come and i love the fact

that it makes me very proud as a father

knowing that many of these experiences

will be us

now the problem is who owns history

there’s a big issue of bureaucracy of

academia right now

i’ve traveled the world and i’ve talked

with many universities

and look i’m not i do not have a

master’s

i have a bachelor of recording arts i’m

an audio engineer guys

but there is a there’s a bit of a bias

going on it’s a bit close

and a lot of waste and a lot of time

there’s no sense of urgency

in these in these entities but the

bigger culprits guys

really they’re the corporations

they’re doing what is could be only said

as a digital land grab right now

and it’s quite disturbing they’re

hoarding massive amounts of data

and the fake pretense saying that they

are going to provide the star to the

public to for

people like myself to do things with it

they’re providing us low resolution

jpegs

when we need the high resolution raw

photography you can google them

and so we believe we are on the right

side of history in this

uh my unique situation allowed us to

have

a fundamentally stronger way of us

because it was a passion project

so machine learning to the rescue

mundane tasks take us so with queen

nefertari for an example

any one of our experiences it takes

three guys six weeks

and about twenty thousand dollars to

produce one of these experiences

which is actually considerably cheaper

than any of the competition and

considerably faster

but we can’t do it at this pace we need

to find better ways

you know i refer to the tasks as pushing

pixels

it’s exhausting but luckily we’re an r d

company and so when deep learning

machine learning came to be a thing

we already had all the computing power

we already had

all the resources we’ve required but

most importantly we had plenty of data

we had tens of you know hundreds of

thousands of photographs that we could

use to train the system

and so why not teach the computer to do

these boring tasks

one of these tasks is uh what we call

contextually intelligent interpretation

and painting basically we’re able to go

and lesser around something we don’t

want in one of these scenes

once again with never tire we had wooden

floorboards we had no smoking signs we

had plaques we had halogen lighting

circle around and the computer knows how

to fill it in with something that is

contextually artistically correct

uh other reasons why this is so useful

is quite frankly because

where we get occlusion issues in our

photogrammetry we don’t take enough

photographs

this is able to look at those low

density areas and fill it in

artistically as good as any human could

do

if not better semantic image

segmentation this is the ability for us

to essentially

optimize and divide our meshes um divide

our environments to work faster on

machines

and also tag them with basic principles

like wooden floors

have these properties uh carpet has

these properties

and this is where it gets really cool

for audio instead of someone having to

manually go in

and label each one of these objects with

certain properties

we can now have it done automatically

and so this creates

way better immersive experiences for

audio because things reflect sound the

way you’d expect them to reflect

noise removal now this is this is like

magic

there’s your image there’s what we get

that’s your base truth and so we’ve been

able to look why this is useful

is not all photographs are equal right

and so we basically have a situation

where we can enhance

greatly and so how do we use this we can

resurrect

archival footage we can resurrect stuff

that did not

was never intended for photogrammetry

and so

uh i was actually recently asked by

time magazine and my good friend peter

martin were in the running

to do martin luther king’s i had a dream

speech use all that old archival footage

and see if we can bring it back to life

um but christchurch cathedral that’s

something that we really have been

actively looking at so we’re looking at

all the old footage

all the old wedding photos we did a big

call to action about a year ago

trying to archive as much data as we can

to see if we can recreate the church and

it’s in its glory and it’s beauty

uh dppr is our most immediate and this

is like live now we have an early alpha

that

people can try out this technology is

able to look at a photograph

and extrapolate it to its true

fundamentals

uh what that means is we can remove

harsh lighting we can move highlights

um it does a whole bunch other

techniques like super sampling and noise

removal as well

but where it is really good is that with

photogrammetry when we go to these

environments we have baked lighting

depending on where the sun is you know

and you don’t want that if you want to

be able to realize an environment you

need to remove it all

and this is a massive deal for the

photogrammetry community but we also

were able to create byproducts of us so

we can a single person can now take a

photo

and essentially produce all the texture

maps they need for any vfx package

or game engine of their choice and this

is

incredible so an individual one guy can

actually do more now

for what a team of 30 could do in three

months

so the need for economies of scale we

have an arsenal of tools

we have mass automation we

this allows us to have economies of

scale and reduce costs dramatically

so we can do more now i can’t be

everywhere

okay as much as i like flying i actually

hate flying but as much as i like

traveling

um we need people on the ground to

actually

acquisition the starter but we need to

create incentives to allow them to want

to do this

and so we’re saying let’s introduce an

artist’s rights management system

where that photograph is

you know is basically secured by a

digital ledger so where that photograph

that individual is tagged

they can say what that photograph is

used for and what use cases

um so it’s think of it like creative

commons in a different level of creative

commons

so we need to make this uh

basically work for everyone and not

cross any lines basically

and also the artist needs to get paid

royalties

this is important so for now

philanthropic entities we believe in the

short term can jump in and fund many of

these endeavors very similar how we did

to antarctica

these great expeditions of the world of

past centuries gone

and these parties were you know to

mention in the history books we know we

know who the heroes were who went to

these

places and we know the people who

bankrolled it there’s nothing wrong with

that

licensing to institutes universities

schools educational facilities the

museology scene museums

they are dying they’re craving for these

kind of experiences and right now

they’re

having to spend hundreds of thousands of

doing it for themselves

and it’s usually closed off we don’t

want that we want everyone to be able to

experience this

now the exciting thing that um i spend a

lot of time in la

uh what film studios are doing now with

real-time virtualization

we’re they’re shooting everything on

green screen now and they’re going out

and getting these photogrammetry

environments and placing people on them

we’re saying look instead of you sending

a team of 20 to do it

license that environment through us and

we’ll provide you a fraction of the cost

so we have a decentralized backup

peer-to-peer none of these photos stay

in one place it’s encrypted

but it’s distributed and we have a

marketplace that benefits everyone

so the conclusion guys an army of

millions

running around taking these photographs

acquisitioning this data

a decentralized secure database that

protects the starter

computing power and ai deep learning

making this an

automated process

and we as a result get

mass real-world encapsulations for

everyone

and this is how we’re going to back up

the planet