What were learning from online education Daphne Koller

like many of you on one of the lucky

people I was born to a family where

education was pervasive I’m a third

generation PhD a daughter of two

academics my childhood I played around

in my father’s university lab so it was

taken for granted that I attend some of

the best universities which in turn

opened the door to a world of

opportunity unfortunately most of the

people in the world are not so lucky in

some parts of the world for example

South Africa education is just not

readily accessible in South Africa the

educational system was constructed in

the days of apartheid for the white

minority and as a consequence today

there’s just not enough spots for the

many more people who want and deserve a

high quality education that scarcity led

to a crisis in January of this year at

the University of Johannes Berg there

were a handful of positions left open

from the sander admissions process and

the night before they were supposed to

open that free registration thousands of

people lined up outside the gates in the

line a mile long hoping to be first in

line to get one of those positions when

the gates opened there was a stampede

and 20 people were injured and one woman

died she was a mother who gave her life

trying to get her son a chance at a

better life but even in parts of the

world like the United States where

education is available it might not be

within reach there’s been much discussed

in the last few years about the rising

cost of health care what might not be

quite as obvious to people is that

during that same period the cost of

higher education tuition have been

increasing it almost twice the rate for

a total of five hundred and fifty nine

percent since 1985 this makes education

unaffordable for many people finally

even for those who do manage to get a

higher education

the doors of opportunity might not open

only a little over half of recent

college graduates in the United States

who get a higher education actually are

working in jobs that require that

education this of course is not true for

the students who graduate from the top

institutions

but for many others they do not get the

value for their time and their efforts

Tom Friedman in his recent New York

Times article captured in the way that

no one else could the spirit behind

their effort he said the big

breakthroughs are what happens when what

is suddenly possible meets what is

desperately necessary I’ve talked about

what’s desperately necessary let’s talk

about what’s suddenly possible what’s

suddenly possible was demonstrated by

three big Stanford classes each of which

had an enrollment of a hundred thousand

people or more so to understand this

let’s look at one of those classes the

machine learning class offered by my

colleague and co-founder and ring andrew

teaches one of the bigger stanford

classes it’s a machine learning class

and insa has 400 people enrolled every

time it’s offered when andrew thought

the machine learning class to the

general public it had a hundred thousand

people registered so to put that number

in perspective for andrew treats that

same sized audience by teaching a

stanford class he would have to do that

for 250 years of course he’d get really

bored so having seen the impact of this

Andrew and I decided that we needed to

really try and scale this up to bring

the best quality education to as many

people as we could so we formed Coursera

whose goal is to take the best courses

from the best instructors at the best

universities and provide it to everyone

around the world for free we currently

have 43 courses on the platform from

four universities across a range of

disciplines and let me show you a little

bit of an overview of what that looks

like

welcome to calculus 50 million people

are uninsured on us help us design more

effective institutions of policy we get

unbelievable segregation so Bush imagine

that in the future you’d wear a camera

right in the center of your head

Mills wants the student of sociology to

develop the quality of mind hanging

cable takes on the form of a hyperbolic

cosine for each pixel in the image set

the red to zero vaccine allowed us to

eliminate polio virus as luke turns us

serve breakfast and San Jose

well that sounds funny so this is which

coin you pick so this is the two pops

observe a large scale machine learning

it turns out maybe not surprisingly the

students like getting the best content

from the best universities for free

since we opened the website in February

we now have six hundred and forty

thousand students from 190 countries we

have 1.5 million Ruhlman six million

quizzes in the 15 classes that have

launched so far have been submitted and

14 million videos have been viewed but

it’s not just about the numbers it’s

also about the people whether it’s a

Kosh who comes from a small town in

India and would never have access in

this case to a Stanford quality course

and would never be able to afford it or

Jenny who was a single mother of two and

wants to hone her skills so that she can

go back and complete her master’s degree

or Ryan who can’t go to school because

his immune deficient daughter can’t be

risk to have germs come into the house

so we couldn’t leave the house I’m

really glad to say recently we’ve been

in correspondence with Ryan that this

story had a happy ending

baby Shannon you can see her on the left

is doing much better now and Ryan got a

job by taking some of our courses so

what made these courses so different

after all online course content has been

available for a while what made it

different is that this was a real course

experience it started on a given day and

then the students would watch videos on

a weekly basis and do homework

assignments and these would be real

homework assignments

for real grade with a real deadline you

can see the deadlines in the usage graph

these are the spikes showing that

procrastination is a global phenomenon

at the end of the course the students

got a certificate they could present

that certificate to a prospective

employer and get a better job and we

know many students who did some students

took their certificate and presented

this to an educational institution at

which they were enrolled for actual

college credits so these students were

really getting something meaningful for

their investment of time and effort

let’s talk a little bit about some of

the components that go into these

courses the first component is that when

you move away from the constraints of a

physical classroom and design content

explicitly for an online format you can

break away from for example the

monolithic one-hour lecture you can

break up the material for example into

these short modular units of 8 to 12

minutes each of which represents a

coherent concept students can traverse

this material in different ways

depending on their background their

skills or their interests so for example

some students might benefit from a

little bit of prepare Tori material that

other students might already have other

students might be interested in a

particular enrichment topic that they

want to pursue individually so this

format allows us to break away from the

one-size-fits-all model of Education and

allow students to follow a much more

personalized curriculum of course we all

know as educators that students don’t

learn by sitting and passively watching

videos perhaps one of the biggest

components of this effort is that we

need to have students who practice with

the material and really understand order

to really understand it there’s been a

range of studies that demonstrate the

importance of this this one that

appeared in science last year for

example demonstrates that even simple

retrieval practice where students are

just supposed to repeat what they

already learned gives considerably

improved results on various achievement

tests down the line than many other

educational interventions we’ve tried to

build in retrieval practice into the

platform as well as other forms of

practice

in many ways for example even our videos

are not just videos every few minutes

the video pauses and the students get

asked a question hyperbolic discounting

status quo bias base rate bias they’re

all well documents so they’re all well

documented deviations from rational

behavior so here are the video pauses

and the student types send the answer

into the box and submits obviously they

weren’t paying attention so they get to

try again and this time they got it

right there’s an optional explanation if

they want and now the video moves on to

the next part is a lecture this is kind

of simple question that I as an

instructor might ask in class but when I

ask that kind of a question in class 80%

of the students are still scribbling the

last thing I said 15% are zoned out on

Facebook and then there is a smarty

pants in the front row who blurts out

the answer before anyone else has had a

chance to think about it and I as the

instructor I’m terribly gratified that

somebody actually knew the answer and so

the lecture moves on before really most

of the students have even noticed that a

question had been asked here every

single student has to engage with the

material now of course these simple

retrieval questions are not the end of

the story one needs to build in much

more meaningful practice questions and

one also needs to provide the students

with feedback on those questions now how

do you grade the work of a hundred

thousand students if you do not have ten

thousand TAS the answer is you need to

abuse technology to do it for you now

fortunately technology has come a long

way and we can now great a range of

interesting types of homework in

addition to multiple-choice and the

kinds of short answer questions that you

saw in the video we can also grade math

mathematical expressions as well as

mathematical derivations we could grade

models whether it’s financial models in

a business class or physical models in a

science or engineering class and we can

grade some pretty sophisticated

programming assignments so let me show

you one that’s actually pretty simple

but fairly visual this is from

Stanford’s computer science 101 class

and the student is supposed to color

correct that blurry red image they’re

typing their program into the browser

and you can see they didn’t get a quite

right

lady liberty still

sic and so the student rise again and

now they got it right and they’re told

back and they can move on to the next

assignment this ability to interact

actively with the material and be told

when you’re right or wrong is really

essential to student learning now of

course we cannot yet grade the range of

work that one needs for all courses

specifically what’s lacking is the kind

of critical thinking work that is so

essential in such disciplines as the

humanities the social sciences business

and others so we tried to convince for

example some of our humanities faculty

that multiple choice was not such a bad

strategy that didn’t go over really well

so we had to come up with a different

solution and the solution we ended up

using is peer grading it turns out that

previous studies show like this one by

Sadler and good the peer grading is a

surprisingly effective strategy for

providing reproducible great it was

tried only in small classes but there it

showed for example that the students per

son grade on the y-axis are actually

very well correlated with a teacher

assigned grade on the x-axis what’s even

more surprising is that self– grades

where the student grade their own work

critically so long as you incentivize

them properly so they can’t give

themselves a perfect score or actually

even better correlated with a teacher

grades and so this is an effective

strategy that can be used for grading at

scale and is also a useful learning

strategy for the students because they

actually learn from the experience so we

now have the largest peer grading

pipeline ever devised where tens of

thousands of students are grading each

other’s work and quite successfully I

have to say but this is not just about

students sitting alone in their living

room working through problems around

each one of our courses a community of

students had formed a global community

of people around a shared intellectual

endeavor what you see here is a self

generated map from students and ours

Princeton sociology 101 course where

they have put themselves in a world map

and you can really see the global reach

of this kind of effort students

collaborated in these courses in a

variety of different ways first of all

there was a question-and-answer forum

where students with posed questions and

other students would answer those

questions

and the really amazing thing is because

there were so many students it means

that even if the student posed the

question at three o’clock in the morning

somewhere around the world there would

be somebody who was awake and working on

the same problem and so in many of our

courses the median response time for a

question on the question and answer form

was twenty-two minutes

which is not a level of service I have

ever offered to my Stanford students and

you can see from the student

testimonials that students actually find

that because of this large online

community they got to interact with each

other in many ways that were deeper than

they did in the context of the physical

classroom students also self-assembled

that any kind of intervention from us

into small study groups some of these

were physical study groups along

geographical constraints and met on a

weekly basis to work through problem

sets this is the San Francisco study

group but there were ones all over the

world

others were virtual study groups

sometimes a long language lines are

along cultural lines and the bottom left

there you see our multi cultural

Universal study group where people

explicitly wanted to connect with people

from other cultures there’s some

tremendous opportunities to be had from

this kind of framework the first is that

it has the potential of giving us a

completely unprecedented look into

understanding human learning because the

data that we can collect here is unique

you can collect every click every

homework submission every forum posts

from tens of thousands of students so

you can turn the study of human learning

from the hypothesis driven mode to the

data driven mode a transformation that

for example has revolutionized biology

you can use these data to understand

fundamental questions like what are good

learning strategies that are effective

versus ones that are not and in the

context of particular courses you can

ask questions like what are some of the

misconceptions that are more common and

how do we help students fix them so here

is an example of that also from Andrews

machine learning class this is a

distribution of wrong answers to one of

Andrews

assignments the answers happen to be

pairs of numbers so you can draw them on

this two-dimensional plot each of the

little crosses that you see is a

different wrong answer the big cross at

the top left is where 2,000 students

gave the exact same wrong answer now

it’s two students in the class of a

hundred give the same wrong answer you

would never notice but when two thousand

students give the same wrong answer it’s

kind of hard to miss so Andrew and his

students went and looked at some of

those assignments understood the the

root cause of the misconception and then

they produced a targeted error message

that would be provided to every student

whose answer fell into that bucket which

means that students who made that same

mistake would now get personalized

feedback telling them how to fix their

misconception much more effectively so

there’s so this personalization is

something that one can then build by

having the virtue of large numbers

personalization is perhaps one of the

biggest opportunities here as well

because it provides us with the

potential of solving a thirty-year-old

problem educational researcher Benjamin

bloom in 1984 posed what’s called the

two sigma problem which he observed by

studying three populations the first is

the population that studied in a lecture

based classroom the second is a

population of students that studied

using a standard lecture based classroom

but with a mastery based approach so the

students couldn’t move on to the next

topic before demonstrating mastery of

the previous one and finally there was a

population of students that were taught

in a one-on-one instruction using a

tutor the mastery based population is a

full standard deviation or Sigma in

achievement scores better than the

centre’s lecture based class and the

individual tutoring gives you 2 Sigma

improvements in performance to

understand what that means let’s look at

the lecture based classroom let’s pick

the median performance as a threshold so

in the lecture based class half the

students are above that level and half

are below in the individual tutoring

instruction 98% of the students are

going to be above that threshold imagine

if we could teach so that 98% of our

students would

above average hence the two sigma

problem because we cannot afford as a

society to provide every student with an

individual human tutor but they we can

afford to provide each student with a

computer or smartphone so the question

is how can we use technology to push

from the left side of the graph from the

blue curve to the right side with the

green curve mastery is easy to achieve

using computer because the computer

doesn’t get tired of showing you the

same video five times and it doesn’t

even get tired of braiding the same work

multiple times we’ve seen that in many

of the examples that I’ve shown you an

even personalization is something that

we’re starting to see the beginnings of

whether it’s via the personalized

trajectory through the curriculum or

some of the personalized feedback that

we’ve shown you so the goal here is to

try and push and see how far we can get

towards the green curve so if this is so

great our universities now obsolete well

Mark Twain certainly thought so

he said the college is a place where

professors lecture notes go straight to

students lecture notes without passing

through the reins of either

I picked a different with Mark Twain

though I think what he was complaining

about is not is not universities but

rather the lecture based format that so

many universities spent so much time on

so let’s go from back even further to

Plutarch who said that the mind is not a

vessel that needs filling but wood that

needs igniting and maybe we should spend

less time at universities filling our

students minds with content by lecturing

at them and more time igniting their

creativity their imagination and their

problem-solving skills by actually

talking with them so how do we do that

we do that by doing active learning in

the classroom so there has been many

studies including this one that show

that if you use active learning

interacting with your students in the

classroom performance improves on every

single metric on attendance on

engagement and on learning as measured

by a standardized test you can see for

example that the achievement score

almost doubles in this particular

experiment so maybe this is how we

should spend our time at universities so

to summarize if we could offer a top

quality education to everyone around the

world for free what would that do three

things first it would establish

education of fundamental human rights

where anyone around the world with the

ability and the motivation it could get

the skills that they need to make a

better life for themselves their

families and their communities

second it would enable lifelong learning

it’s a shame that for so many people

learning stops when we finish high

school or when we finished college by

having this amazing content be available

we would be able to learn something new

every time you wanted whether it’s just

to expand their minds or if to change

our lives

and finally this would enable a wave of

innovation because amazing talent can be

found anywhere maybe the next Albert

Einstein or the next Steve Jobs is

living somewhere in a remote village in

Africa and if we could offer that person

an education they would be able to come

up with the next big idea and make the

world a better place for all of us thank

you very much

you