What were learning from online education Daphne Koller
like many of you on one of the lucky
people I was born to a family where
education was pervasive I’m a third
generation PhD a daughter of two
academics my childhood I played around
in my father’s university lab so it was
taken for granted that I attend some of
the best universities which in turn
opened the door to a world of
opportunity unfortunately most of the
people in the world are not so lucky in
some parts of the world for example
South Africa education is just not
readily accessible in South Africa the
educational system was constructed in
the days of apartheid for the white
minority and as a consequence today
there’s just not enough spots for the
many more people who want and deserve a
high quality education that scarcity led
to a crisis in January of this year at
the University of Johannes Berg there
were a handful of positions left open
from the sander admissions process and
the night before they were supposed to
open that free registration thousands of
people lined up outside the gates in the
line a mile long hoping to be first in
line to get one of those positions when
the gates opened there was a stampede
and 20 people were injured and one woman
died she was a mother who gave her life
trying to get her son a chance at a
better life but even in parts of the
world like the United States where
education is available it might not be
within reach there’s been much discussed
in the last few years about the rising
cost of health care what might not be
quite as obvious to people is that
during that same period the cost of
higher education tuition have been
increasing it almost twice the rate for
a total of five hundred and fifty nine
percent since 1985 this makes education
unaffordable for many people finally
even for those who do manage to get a
higher education
the doors of opportunity might not open
only a little over half of recent
college graduates in the United States
who get a higher education actually are
working in jobs that require that
education this of course is not true for
the students who graduate from the top
institutions
but for many others they do not get the
value for their time and their efforts
Tom Friedman in his recent New York
Times article captured in the way that
no one else could the spirit behind
their effort he said the big
breakthroughs are what happens when what
is suddenly possible meets what is
desperately necessary I’ve talked about
what’s desperately necessary let’s talk
about what’s suddenly possible what’s
suddenly possible was demonstrated by
three big Stanford classes each of which
had an enrollment of a hundred thousand
people or more so to understand this
let’s look at one of those classes the
machine learning class offered by my
colleague and co-founder and ring andrew
teaches one of the bigger stanford
classes it’s a machine learning class
and insa has 400 people enrolled every
time it’s offered when andrew thought
the machine learning class to the
general public it had a hundred thousand
people registered so to put that number
in perspective for andrew treats that
same sized audience by teaching a
stanford class he would have to do that
for 250 years of course he’d get really
bored so having seen the impact of this
Andrew and I decided that we needed to
really try and scale this up to bring
the best quality education to as many
people as we could so we formed Coursera
whose goal is to take the best courses
from the best instructors at the best
universities and provide it to everyone
around the world for free we currently
have 43 courses on the platform from
four universities across a range of
disciplines and let me show you a little
bit of an overview of what that looks
like
welcome to calculus 50 million people
are uninsured on us help us design more
effective institutions of policy we get
unbelievable segregation so Bush imagine
that in the future you’d wear a camera
right in the center of your head
Mills wants the student of sociology to
develop the quality of mind hanging
cable takes on the form of a hyperbolic
cosine for each pixel in the image set
the red to zero vaccine allowed us to
eliminate polio virus as luke turns us
serve breakfast and San Jose
well that sounds funny so this is which
coin you pick so this is the two pops
observe a large scale machine learning
it turns out maybe not surprisingly the
students like getting the best content
from the best universities for free
since we opened the website in February
we now have six hundred and forty
thousand students from 190 countries we
have 1.5 million Ruhlman six million
quizzes in the 15 classes that have
launched so far have been submitted and
14 million videos have been viewed but
it’s not just about the numbers it’s
also about the people whether it’s a
Kosh who comes from a small town in
India and would never have access in
this case to a Stanford quality course
and would never be able to afford it or
Jenny who was a single mother of two and
wants to hone her skills so that she can
go back and complete her master’s degree
or Ryan who can’t go to school because
his immune deficient daughter can’t be
risk to have germs come into the house
so we couldn’t leave the house I’m
really glad to say recently we’ve been
in correspondence with Ryan that this
story had a happy ending
baby Shannon you can see her on the left
is doing much better now and Ryan got a
job by taking some of our courses so
what made these courses so different
after all online course content has been
available for a while what made it
different is that this was a real course
experience it started on a given day and
then the students would watch videos on
a weekly basis and do homework
assignments and these would be real
homework assignments
for real grade with a real deadline you
can see the deadlines in the usage graph
these are the spikes showing that
procrastination is a global phenomenon
at the end of the course the students
got a certificate they could present
that certificate to a prospective
employer and get a better job and we
know many students who did some students
took their certificate and presented
this to an educational institution at
which they were enrolled for actual
college credits so these students were
really getting something meaningful for
their investment of time and effort
let’s talk a little bit about some of
the components that go into these
courses the first component is that when
you move away from the constraints of a
physical classroom and design content
explicitly for an online format you can
break away from for example the
monolithic one-hour lecture you can
break up the material for example into
these short modular units of 8 to 12
minutes each of which represents a
coherent concept students can traverse
this material in different ways
depending on their background their
skills or their interests so for example
some students might benefit from a
little bit of prepare Tori material that
other students might already have other
students might be interested in a
particular enrichment topic that they
want to pursue individually so this
format allows us to break away from the
one-size-fits-all model of Education and
allow students to follow a much more
personalized curriculum of course we all
know as educators that students don’t
learn by sitting and passively watching
videos perhaps one of the biggest
components of this effort is that we
need to have students who practice with
the material and really understand order
to really understand it there’s been a
range of studies that demonstrate the
importance of this this one that
appeared in science last year for
example demonstrates that even simple
retrieval practice where students are
just supposed to repeat what they
already learned gives considerably
improved results on various achievement
tests down the line than many other
educational interventions we’ve tried to
build in retrieval practice into the
platform as well as other forms of
practice
in many ways for example even our videos
are not just videos every few minutes
the video pauses and the students get
asked a question hyperbolic discounting
status quo bias base rate bias they’re
all well documents so they’re all well
documented deviations from rational
behavior so here are the video pauses
and the student types send the answer
into the box and submits obviously they
weren’t paying attention so they get to
try again and this time they got it
right there’s an optional explanation if
they want and now the video moves on to
the next part is a lecture this is kind
of simple question that I as an
instructor might ask in class but when I
ask that kind of a question in class 80%
of the students are still scribbling the
last thing I said 15% are zoned out on
Facebook and then there is a smarty
pants in the front row who blurts out
the answer before anyone else has had a
chance to think about it and I as the
instructor I’m terribly gratified that
somebody actually knew the answer and so
the lecture moves on before really most
of the students have even noticed that a
question had been asked here every
single student has to engage with the
material now of course these simple
retrieval questions are not the end of
the story one needs to build in much
more meaningful practice questions and
one also needs to provide the students
with feedback on those questions now how
do you grade the work of a hundred
thousand students if you do not have ten
thousand TAS the answer is you need to
abuse technology to do it for you now
fortunately technology has come a long
way and we can now great a range of
interesting types of homework in
addition to multiple-choice and the
kinds of short answer questions that you
saw in the video we can also grade math
mathematical expressions as well as
mathematical derivations we could grade
models whether it’s financial models in
a business class or physical models in a
science or engineering class and we can
grade some pretty sophisticated
programming assignments so let me show
you one that’s actually pretty simple
but fairly visual this is from
Stanford’s computer science 101 class
and the student is supposed to color
correct that blurry red image they’re
typing their program into the browser
and you can see they didn’t get a quite
right
lady liberty still
sic and so the student rise again and
now they got it right and they’re told
back and they can move on to the next
assignment this ability to interact
actively with the material and be told
when you’re right or wrong is really
essential to student learning now of
course we cannot yet grade the range of
work that one needs for all courses
specifically what’s lacking is the kind
of critical thinking work that is so
essential in such disciplines as the
humanities the social sciences business
and others so we tried to convince for
example some of our humanities faculty
that multiple choice was not such a bad
strategy that didn’t go over really well
so we had to come up with a different
solution and the solution we ended up
using is peer grading it turns out that
previous studies show like this one by
Sadler and good the peer grading is a
surprisingly effective strategy for
providing reproducible great it was
tried only in small classes but there it
showed for example that the students per
son grade on the y-axis are actually
very well correlated with a teacher
assigned grade on the x-axis what’s even
more surprising is that self– grades
where the student grade their own work
critically so long as you incentivize
them properly so they can’t give
themselves a perfect score or actually
even better correlated with a teacher
grades and so this is an effective
strategy that can be used for grading at
scale and is also a useful learning
strategy for the students because they
actually learn from the experience so we
now have the largest peer grading
pipeline ever devised where tens of
thousands of students are grading each
other’s work and quite successfully I
have to say but this is not just about
students sitting alone in their living
room working through problems around
each one of our courses a community of
students had formed a global community
of people around a shared intellectual
endeavor what you see here is a self
generated map from students and ours
Princeton sociology 101 course where
they have put themselves in a world map
and you can really see the global reach
of this kind of effort students
collaborated in these courses in a
variety of different ways first of all
there was a question-and-answer forum
where students with posed questions and
other students would answer those
questions
and the really amazing thing is because
there were so many students it means
that even if the student posed the
question at three o’clock in the morning
somewhere around the world there would
be somebody who was awake and working on
the same problem and so in many of our
courses the median response time for a
question on the question and answer form
was twenty-two minutes
which is not a level of service I have
ever offered to my Stanford students and
you can see from the student
testimonials that students actually find
that because of this large online
community they got to interact with each
other in many ways that were deeper than
they did in the context of the physical
classroom students also self-assembled
that any kind of intervention from us
into small study groups some of these
were physical study groups along
geographical constraints and met on a
weekly basis to work through problem
sets this is the San Francisco study
group but there were ones all over the
world
others were virtual study groups
sometimes a long language lines are
along cultural lines and the bottom left
there you see our multi cultural
Universal study group where people
explicitly wanted to connect with people
from other cultures there’s some
tremendous opportunities to be had from
this kind of framework the first is that
it has the potential of giving us a
completely unprecedented look into
understanding human learning because the
data that we can collect here is unique
you can collect every click every
homework submission every forum posts
from tens of thousands of students so
you can turn the study of human learning
from the hypothesis driven mode to the
data driven mode a transformation that
for example has revolutionized biology
you can use these data to understand
fundamental questions like what are good
learning strategies that are effective
versus ones that are not and in the
context of particular courses you can
ask questions like what are some of the
misconceptions that are more common and
how do we help students fix them so here
is an example of that also from Andrews
machine learning class this is a
distribution of wrong answers to one of
Andrews
assignments the answers happen to be
pairs of numbers so you can draw them on
this two-dimensional plot each of the
little crosses that you see is a
different wrong answer the big cross at
the top left is where 2,000 students
gave the exact same wrong answer now
it’s two students in the class of a
hundred give the same wrong answer you
would never notice but when two thousand
students give the same wrong answer it’s
kind of hard to miss so Andrew and his
students went and looked at some of
those assignments understood the the
root cause of the misconception and then
they produced a targeted error message
that would be provided to every student
whose answer fell into that bucket which
means that students who made that same
mistake would now get personalized
feedback telling them how to fix their
misconception much more effectively so
there’s so this personalization is
something that one can then build by
having the virtue of large numbers
personalization is perhaps one of the
biggest opportunities here as well
because it provides us with the
potential of solving a thirty-year-old
problem educational researcher Benjamin
bloom in 1984 posed what’s called the
two sigma problem which he observed by
studying three populations the first is
the population that studied in a lecture
based classroom the second is a
population of students that studied
using a standard lecture based classroom
but with a mastery based approach so the
students couldn’t move on to the next
topic before demonstrating mastery of
the previous one and finally there was a
population of students that were taught
in a one-on-one instruction using a
tutor the mastery based population is a
full standard deviation or Sigma in
achievement scores better than the
centre’s lecture based class and the
individual tutoring gives you 2 Sigma
improvements in performance to
understand what that means let’s look at
the lecture based classroom let’s pick
the median performance as a threshold so
in the lecture based class half the
students are above that level and half
are below in the individual tutoring
instruction 98% of the students are
going to be above that threshold imagine
if we could teach so that 98% of our
students would
above average hence the two sigma
problem because we cannot afford as a
society to provide every student with an
individual human tutor but they we can
afford to provide each student with a
computer or smartphone so the question
is how can we use technology to push
from the left side of the graph from the
blue curve to the right side with the
green curve mastery is easy to achieve
using computer because the computer
doesn’t get tired of showing you the
same video five times and it doesn’t
even get tired of braiding the same work
multiple times we’ve seen that in many
of the examples that I’ve shown you an
even personalization is something that
we’re starting to see the beginnings of
whether it’s via the personalized
trajectory through the curriculum or
some of the personalized feedback that
we’ve shown you so the goal here is to
try and push and see how far we can get
towards the green curve so if this is so
great our universities now obsolete well
Mark Twain certainly thought so
he said the college is a place where
professors lecture notes go straight to
students lecture notes without passing
through the reins of either
I picked a different with Mark Twain
though I think what he was complaining
about is not is not universities but
rather the lecture based format that so
many universities spent so much time on
so let’s go from back even further to
Plutarch who said that the mind is not a
vessel that needs filling but wood that
needs igniting and maybe we should spend
less time at universities filling our
students minds with content by lecturing
at them and more time igniting their
creativity their imagination and their
problem-solving skills by actually
talking with them so how do we do that
we do that by doing active learning in
the classroom so there has been many
studies including this one that show
that if you use active learning
interacting with your students in the
classroom performance improves on every
single metric on attendance on
engagement and on learning as measured
by a standardized test you can see for
example that the achievement score
almost doubles in this particular
experiment so maybe this is how we
should spend our time at universities so
to summarize if we could offer a top
quality education to everyone around the
world for free what would that do three
things first it would establish
education of fundamental human rights
where anyone around the world with the
ability and the motivation it could get
the skills that they need to make a
better life for themselves their
families and their communities
second it would enable lifelong learning
it’s a shame that for so many people
learning stops when we finish high
school or when we finished college by
having this amazing content be available
we would be able to learn something new
every time you wanted whether it’s just
to expand their minds or if to change
our lives
and finally this would enable a wave of
innovation because amazing talent can be
found anywhere maybe the next Albert
Einstein or the next Steve Jobs is
living somewhere in a remote village in
Africa and if we could offer that person
an education they would be able to come
up with the next big idea and make the
world a better place for all of us thank
you very much
you