[Script Info]
Title:
[Events]
Format: Layer, Start, End, Style, Name, MarginL, MarginR, MarginV, Effect, Text
Dialogue: 0,0:00:09.63,0:00:12.55,Default,,0000,0000,0000,,so we've been talking about information
Dialogue: 0,0:00:12.61,0:00:15.78,Default,,0000,0000,0000,,how you measure information
Dialogue: 0,0:00:19.33,0:00:20.94,Default,,0000,0000,0000,,and of course, you measure information in bits
Dialogue: 0,0:00:21.75,0:00:26.87,Default,,0000,0000,0000,,how you can use information to label things
Dialogue: 0,0:00:27.35,0:00:30.71,Default,,0000,0000,0000,,for instance, with bar codes
Dialogue: 0,0:00:34.80,0:00:37.03,Default,,0000,0000,0000,,so you can use information
Dialogue: 0,0:00:37.06,0:00:38.07,Default,,0000,0000,0000,,to label things
Dialogue: 0,0:00:38.07,0:00:40.15,Default,,0000,0000,0000,,and then we talked about
Dialogue: 0,0:00:40.48,0:00:45.49,Default,,0000,0000,0000,,probability and information
Dialogue: 0,0:00:45.49,0:00:47.25,Default,,0000,0000,0000,,and if I have probabilities for events P_i
Dialogue: 0,0:00:47.25,0:00:51.58,Default,,0000,0000,0000,,then it says that the amount of information
Dialogue: 0,0:00:51.75,0:00:53.53,Default,,0000,0000,0000,,that's associated with this event occurring is
Dialogue: 0,0:00:54.01,0:00:56.16,Default,,0000,0000,0000,,minus the sum over i
Dialogue: 0,0:00:56.43,0:00:58.26,Default,,0000,0000,0000,,p_i log to the base 2
Dialogue: 0,0:00:58.33,0:01:00.83,Default,,0000,0000,0000,,of p_i
Dialogue: 0,0:01:01.16,0:01:02.05,Default,,0000,0000,0000,,this beautiful formula that was developed
Dialogue: 0,0:01:02.51,0:01:04.03,Default,,0000,0000,0000,,by Maxwell, Boltzmann, and Gibbs
Dialogue: 0,0:01:04.73,0:01:07.71,Default,,0000,0000,0000,,back in the middle of nineteenth century
Dialogue: 0,0:01:11.24,0:01:12.11,Default,,0000,0000,0000,,to talk about the amount of entropy
Dialogue: 0,0:01:14.76,0:01:15.03,Default,,0000,0000,0000,,and atoms or molecules
Dialogue: 0,0:01:16.80,0:01:17.05,Default,,0000,0000,0000,,this is often called S
Dialogue: 0,0:01:19.61,0:01:20.53,Default,,0000,0000,0000,,for entropy, as well
Dialogue: 0,0:01:21.60,0:01:22.76,Default,,0000,0000,0000,,and then was rediscovered
Dialogue: 0,0:01:22.82,0:01:23.67,Default,,0000,0000,0000,,by Claude Shannon
Dialogue: 0,0:01:23.74,0:01:24.62,Default,,0000,0000,0000,,in the 1940's
Dialogue: 0,0:01:24.77,0:01:26.59,Default,,0000,0000,0000,,to talk about information theory in the abstract
Dialogue: 0,0:01:26.66,0:01:27.82,Default,,0000,0000,0000,,and the mathematical theory of communication
Dialogue: 0,0:01:28.05,0:01:33.01,Default,,0000,0000,0000,,in fact, there is a funny story
Dialogue: 0,0:01:33.10,0:01:34.76,Default,,0000,0000,0000,,about this that
Dialogue: 0,0:01:35.68,0:01:38.38,Default,,0000,0000,0000,,Shannon, when he came up with this formula
Dialogue: 0,0:01:38.39,0:01:41.14,Default,,0000,0000,0000,,minus sum over i p_i log to the base 2 of p_i
Dialogue: 0,0:01:41.23,0:01:43.23,Default,,0000,0000,0000,,he went to John Von Neumann
Dialogue: 0,0:01:43.42,0:01:45.01,Default,,0000,0000,0000,,the famous mathematician
Dialogue: 0,0:01:45.19,0:01:48.28,Default,,0000,0000,0000,,and he said, "what sould I call this quantity?"
Dialogue: 0,0:01:48.38,0:01:49.82,Default,,0000,0000,0000,,and Von Neumann says
Dialogue: 0,0:01:49.99,0:01:53.64,Default,,0000,0000,0000,,"you should call it H, because that's what Boltzmann called it "
Dialogue: 0,0:01:53.91,0:01:55.45,Default,,0000,0000,0000,,but Von Neumann, who had a famous memory
Dialogue: 0,0:01:57.53,0:01:57.78,Default,,0000,0000,0000,,apparently forgot
Dialogue: 0,0:01:57.78,0:01:59.85,Default,,0000,0000,0000,,of Boltzmann's H
Dialogue: 0,0:02:00.05,0:02:02.08,Default,,0000,0000,0000,,of his famous "H theorem"
Dialogue: 0,0:02:02.35,0:02:03.98,Default,,0000,0000,0000,,was the same thing but without the minus sign
Dialogue: 0,0:02:04.03,0:02:05.21,Default,,0000,0000,0000,,so it's a negative quantity
Dialogue: 0,0:02:05.91,0:02:06.37,Default,,0000,0000,0000,,and it gets more and more negative
Dialogue: 0,0:02:06.55,0:02:07.55,Default,,0000,0000,0000,,as opposed to entropy
Dialogue: 0,0:02:07.58,0:02:08.75,Default,,0000,0000,0000,,which is a positive quantiy
Dialogue: 0,0:02:08.90,0:02:10.55,Default,,0000,0000,0000,,and gets more and more positive
Dialogue: 0,0:02:10.68,0:02:13.17,Default,,0000,0000,0000,,so, actually these fundamental formulas
Dialogue: 0,0:02:13.25,0:02:14.78,Default,,0000,0000,0000,,about information theory
Dialogue: 0,0:02:14.84,0:02:17.37,Default,,0000,0000,0000,,go back to the mid 19th century
Dialogue: 0,0:02:20.24,0:02:20.84,Default,,0000,0000,0000,,a hundred and fifty years
Dialogue: 0,0:02:21.30,0:02:22.63,Default,,0000,0000,0000,,so now we'd like to apply them
Dialogue: 0,0:02:22.94,0:02:25.07,Default,,0000,0000,0000,,to ideas about communication
Dialogue: 0,0:02:25.32,0:02:26.38,Default,,0000,0000,0000,,and to do that, I'd like to tell you
Dialogue: 0,0:02:26.58,0:02:27.58,Default,,0000,0000,0000,,a little bit more about
Dialogue: 0,0:02:27.60,0:02:28.55,Default,,0000,0000,0000,,probability
Dialogue: 0,0:02:32.19,0:02:33.96,Default,,0000,0000,0000,,so, we talked about
Dialogue: 0,0:02:34.43,0:02:34.97,Default,,0000,0000,0000,,probabilities for events
Dialogue: 0,0:02:35.45,0:02:35.82,Default,,0000,0000,0000,,probability x
Dialogue: 0,0:02:37.71,0:02:39.14,Default,,0000,0000,0000,,you know x equals
Dialogue: 0,0:02:42.85,0:02:43.10,Default,,0000,0000,0000,,"it's sunny"
Dialogue: 0,0:02:43.10,0:02:44.61,Default,,0000,0000,0000,,probability of y
Dialogue: 0,0:02:45.66,0:02:49.09,Default,,0000,0000,0000,,y is "it's raining"
Dialogue: 0,0:02:52.79,0:02:53.07,Default,,0000,0000,0000,,and we could look at the probability
Dialogue: 0,0:02:53.07,0:02:54.69,Default,,0000,0000,0000,,of x
Dialogue: 0,0:02:57.03,0:02:57.28,Default,,0000,0000,0000,,and I'm gonna use the notation
Dialogue: 0,0:02:57.28,0:02:59.13,Default,,0000,0000,0000,,I introduced for Boolean logic before
Dialogue: 0,0:03:02.55,0:03:02.80,Default,,0000,0000,0000,,is the probabiity of this thing right here
Dialogue: 0,0:03:04.84,0:03:05.09,Default,,0000,0000,0000,,means "AND"
Dialogue: 0,0:03:05.09,0:03:07.74,Default,,0000,0000,0000,,"probability of X AND Y"
Dialogue: 0,0:03:07.84,0:03:09.48,Default,,0000,0000,0000,,or we can also just call
Dialogue: 0,0:03:13.84,0:03:14.27,Default,,0000,0000,0000,,this the probability of X Y simultaneously
Dialogue: 0,0:03:14.38,0:03:16.100,Default,,0000,0000,0000,,keep on streamlining our notation
Dialogue: 0,0:03:22.52,0:03:23.55,Default,,0000,0000,0000,,this is the probability
Dialogue: 0,0:03:23.70,0:03:32.04,Default,,0000,0000,0000,,that it's raining... it's sunny and it's raining
Dialogue: 0,0:03:36.24,0:03:36.49,Default,,0000,0000,0000,,now, mostly in the world \N
Dialogue: 0,0:03:36.49,0:03:36.88,Default,,0000,0000,0000,,this is a pretty small probability
Dialogue: 0,0:03:38.87,0:03:39.12,Default,,0000,0000,0000,,but here in Santa Fe
Dialogue: 0,0:03:39.77,0:03:40.39,Default,,0000,0000,0000,,it happens all the time
Dialogue: 0,0:03:41.98,0:03:42.23,Default,,0000,0000,0000,,and as a result, you get rather beautiful
Dialogue: 0,0:03:42.23,0:03:43.92,Default,,0000,0000,0000,,rainbows...single, double, triple
Dialogue: 0,0:03:44.07,0:03:45.85,Default,,0000,0000,0000,,on a daily basis
Dialogue: 0,0:03:45.93,0:03:46.79,Default,,0000,0000,0000,,so, we have...
Dialogue: 0,0:03:47.32,0:03:49.64,Default,,0000,0000,0000,,this is what is called
Dialogue: 0,0:03:49.67,0:03:54.26,Default,,0000,0000,0000,,the joint probability
Dialogue: 0,0:03:58.70,0:04:01.86,Default,,0000,0000,0000,,the joint probability that it's sunny and its raining
Dialogue: 0,0:04:01.94,0:04:05.84,Default,,0000,0000,0000,,the joint probability of X and Y
Dialogue: 0,0:04:08.72,0:04:12.80,Default,,0000,0000,0000,,and what do we expect of this\Njoint probability?
Dialogue: 0,0:04:13.43,0:04:16.56,Default,,0000,0000,0000,,so, we have the probability of X and Y
Dialogue: 0,0:04:17.21,0:04:22.19,Default,,0000,0000,0000,,and this tells you the probability \Nthat it's sunny and it's raining
Dialogue: 0,0:04:22.94,0:04:25.00,Default,,0000,0000,0000,,we can also look at the probability \NX AND NOT Y
Dialogue: 0,0:04:34.66,0:04:34.91,Default,,0000,0000,0000,,so X AND (NOT Y)
Dialogue: 0,0:04:35.32,0:04:36.18,Default,,0000,0000,0000,,again using our notation
Dialogue: 0,0:04:39.56,0:04:41.21,Default,,0000,0000,0000,,introduced to us by the famous husband
Dialogue: 0,0:04:41.31,0:04:41.88,Default,,0000,0000,0000,,of the daughter of the severe general \Nof the British ___(?)
Dialogue: 0,0:04:46.56,0:04:46.87,Default,,0000,0000,0000,,George Boole, married to Mary Everest
Dialogue: 0,0:04:46.87,0:04:49.54,Default,,0000,0000,0000,,and we have a relationship which says
Dialogue: 0,0:04:52.12,0:04:52.52,Default,,0000,0000,0000,,that the probability of X on its own
Dialogue: 0,0:04:56.20,0:04:56.45,Default,,0000,0000,0000,,should be equal to the probability \Nof X AND Y plus the\N
Dialogue: 0,0:04:56.45,0:04:59.41,Default,,0000,0000,0000,,probability of X AND (NOT Y)
Dialogue: 0,0:05:00.22,0:05:02.45,Default,,0000,0000,0000,,and the probability of X on its own
Dialogue: 0,0:05:03.08,0:05:07.73,Default,,0000,0000,0000,,is called the "marginal probability"
Dialogue: 0,0:05:10.86,0:05:11.36,Default,,0000,0000,0000,,so, it's just the probability that
Dialogue: 0,0:05:13.02,0:05:13.27,Default,,0000,0000,0000,,it's sunny on its own
Dialogue: 0,0:05:13.27,0:05:15.31,Default,,0000,0000,0000,,so the probability that it's sunny on its own
Dialogue: 0,0:05:15.57,0:05:18.05,Default,,0000,0000,0000,,is the probability that it's sunny and it's raning
Dialogue: 0,0:05:19.53,0:05:19.94,Default,,0000,0000,0000,,plus the probability that it's sunny and it's not raining
Dialogue: 0,0:05:22.20,0:05:22.45,Default,,0000,0000,0000,,I think this makes some kind of sense
Dialogue: 0,0:05:24.48,0:05:24.73,Default,,0000,0000,0000,,why is called the "marginal probability"?
Dialogue: 0,0:05:25.38,0:05:25.63,Default,,0000,0000,0000,,I have no idea
Dialogue: 0,0:05:25.63,0:05:28.43,Default,,0000,0000,0000,,so let's not even worry about it
Dialogue: 0,0:05:28.79,0:05:31.56,Default,,0000,0000,0000,,there's a very nice picture of probabilities
Dialogue: 0,0:05:31.56,0:05:38.98,Default,,0000,0000,0000,,in terms of set theory
Dialogue: 0,0:05:38.98,0:05:41.09,Default,,0000,0000,0000,,I don't know about you
Dialogue: 0,0:05:41.09,0:05:43.44,Default,,0000,0000,0000,,but I grew up in the age of "new math"
Dialogue: 0,0:05:43.44,0:05:44.81,Default,,0000,0000,0000,,where they tried to teach us
Dialogue: 0,0:05:44.81,0:05:46.58,Default,,0000,0000,0000,,about set theory
Dialogue: 0,0:05:46.58,0:05:48.02,Default,,0000,0000,0000,,and unions of sets
Dialogue: 0,0:05:48.02,0:05:50.45,Default,,0000,0000,0000,,and intersections of sets and things like that
Dialogue: 0,0:05:50.45,0:05:53.00,Default,,0000,0000,0000,,from starting at a very early age
Dialogue: 0,0:05:53.00,0:05:54.90,Default,,0000,0000,0000,,which means people of my generation
Dialogue: 0,0:05:54.90,0:05:57.58,Default,,0000,0000,0000,,are completely unable to do their tax returns
Dialogue: 0,0:05:57.58,0:06:00.07,Default,,0000,0000,0000,,but for me, dealing a lot with math
Dialogue: 0,0:06:00.07,0:06:02.26,Default,,0000,0000,0000,,it actually has been quite helpful
Dialogue: 0,0:06:02.26,0:06:04.46,Default,,0000,0000,0000,,for my career to learn about set theory at the age of 3 or 4
Dialogue: 0,0:06:04.46,0:06:06.33,Default,,0000,0000,0000,,or whatever it was
Dialogue: 0,0:06:06.33,0:06:09.26,Default,,0000,0000,0000,,so, we have a picture like this
Dialogue: 0,0:06:09.26,0:06:17.87,Default,,0000,0000,0000,,this is the space or the set of all events
Dialogue: 0,0:06:17.87,0:06:20.57,Default,,0000,0000,0000,,here is the set X
Dialogue: 0,0:06:20.57,0:06:22.31,Default,,0000,0000,0000,,which is the set of events X, where \Nit's sunny
Dialogue: 0,0:06:22.31,0:06:28.35,Default,,0000,0000,0000,,here is the set of events Y, where is \Nthe set of events where it's raining
Dialogue: 0,0:06:30.53,0:06:32.43,Default,,0000,0000,0000,,this thing right here is called
Dialogue: 0,0:06:32.49,0:06:34.08,Default,,0000,0000,0000,,"X intersection Y"
Dialogue: 0,0:06:34.100,0:06:37.30,Default,,0000,0000,0000,,which is the set of events
Dialogue: 0,0:06:37.30,0:06:40.77,Default,,0000,0000,0000,,where it's both sunny and it's raining
Dialogue: 0,0:06:40.77,0:06:43.17,Default,,0000,0000,0000,,but in contrast, if I look at
Dialogue: 0,0:06:43.17,0:06:44.87,Default,,0000,0000,0000,,this right here
Dialogue: 0,0:06:44.87,0:06:47.73,Default,,0000,0000,0000,,this is "X union Y"
Dialogue: 0,0:06:47.73,0:06:49.09,Default,,0000,0000,0000,,which is the set of events
Dialogue: 0,0:06:49.09,0:06:51.66,Default,,0000,0000,0000,,where it's either sunny or raining
Dialogue: 0,0:06:51.66,0:06:52.81,Default,,0000,0000,0000,,and now you can kind of see
Dialogue: 0,0:06:52.81,0:06:59.16,Default,,0000,0000,0000,,where George Boole got his funny\N"cap" and "cup" notation
Dialogue: 0,0:06:59.16,0:07:02.20,Default,,0000,0000,0000,,we can pair this with X AND Y
Dialogue: 0,0:07:02.20,0:07:04.82,Default,,0000,0000,0000,,X AND Y, from a logical standpoint
Dialogue: 0,0:07:04.82,0:07:10.09,Default,,0000,0000,0000,,is essentially the same as this union\Nof these sets
Dialogue: 0,0:07:10.09,0:07:15.04,Default,,0000,0000,0000,,and similarly, X intersection Y \Nis X OR Y --translator's note: professor Lloyd meant "union" when referring to OR and "intersection" when referring to AND http://www.onlinemathlearning.com/intersection-of-two-sets.html--
Dialogue: 0,0:07:15.55,0:07:20.04,Default,,0000,0000,0000,,so when I take the logical statement \Ncorresponding to the set of events
Dialogue: 0,0:07:20.04,0:07:22.20,Default,,0000,0000,0000,,that I write it as X AND Y
Dialogue: 0,0:07:22.20,0:07:26.74,Default,,0000,0000,0000,,the set of events is the intersection \Nof it's sunny and it's raining
Dialogue: 0,0:07:26.74,0:07:32.61,Default,,0000,0000,0000,,X OR Y is the intersection of events \Nwhere it's sunny or it's raining \N--translator's note: professor Lloyd meant "union" \Nwhen referring to OR, "intersection" refers to AND--
Dialogue: 0,0:07:33.12,0:07:37.12,Default,,0000,0000,0000,,and you can have all kinds of you know\Nnice pictures
Dialogue: 0,0:07:42.09,0:07:46.78,Default,,0000,0000,0000,,here's Z where let's say it's snowy at the \Nsame time it's sunny
Dialogue: 0,0:07:46.78,0:07:49.18,Default,,0000,0000,0000,,which is something that I've seen happen\Nhere in Santa Fe
Dialogue: 0,0:07:49.18,0:07:50.07,Default,,0000,0000,0000,,this is not so strange in here
Dialogue: 0,0:07:50.07,0:07:54.81,Default,,0000,0000,0000,,where we have X intersection Y intersection Z
Dialogue: 0,0:07:54.81,0:07:58.90,Default,,0000,0000,0000,,which is not the empty when in terms of Santa Fe
Dialogue: 0,0:07:58.90,0:08:01.34,Default,,0000,0000,0000,,ok, so now let's actually look
Dialogue: 0,0:08:01.34,0:08:02.76,Default,,0000,0000,0000,,at the kinds of information that are\Nassociated with this
Dialogue: 0,0:08:02.76,0:08:10.59,Default,,0000,0000,0000,,suppose that I have a set of possible\Nevents, I'll call one set labeled by i
Dialogue: 0,0:08:10.59,0:08:16.18,Default,,0000,0000,0000,,the other set, labeled by j
Dialogue: 0,0:08:16.18,0:08:22.41,Default,,0000,0000,0000,,and now I can look at p of i and j
Dialogue: 0,0:08:22.41,0:08:25.98,Default,,0000,0000,0000,,so this is a case where the\Nfirst type of event
Dialogue: 0,0:08:25.98,0:08:29.52,Default,,0000,0000,0000,,is i and the second type of event is j
Dialogue: 0,0:08:29.52,0:08:31.73,Default,,0000,0000,0000,,and I can define
Dialogue: 0,0:08:31.73,0:08:33.59,Default,,0000,0000,0000,,you know, I'm gonna do this \Nslightly different
Dialogue: 0,0:08:33.59,0:08:37.18,Default,,0000,0000,0000,,let's call this... we'll be slightly fancier
Dialogue: 0,0:08:37.18,0:08:41.55,Default,,0000,0000,0000,,we'll call these event x_i and event y_j
Dialogue: 0,0:08:41.55,0:08:43.78,Default,,0000,0000,0000,,so, i labels the different events of x \N
Dialogue: 0,0:08:43.78,0:08:46.11,Default,,0000,0000,0000,,and j labels the different events of y
Dialogue: 0,0:08:46.11,0:08:51.49,Default,,0000,0000,0000,,so, for instance x_i could be two events\Neither it's sunny or it's not sunny
Dialogue: 0,0:08:51.49,0:08:55.64,Default,,0000,0000,0000,,so i could be zero, and it would be \N'it's not sunny'
Dialogue: 0,0:08:55.64,0:08:56.54,Default,,0000,0000,0000,,and 1 could be it's sunny
Dialogue: 0,0:08:56.54,0:08:58.35,Default,,0000,0000,0000,,and j could be it's either raining
Dialogue: 0,0:08:58.35,0:08:59.56,Default,,0000,0000,0000,,or it's not raining
Dialogue: 0,0:08:59.56,0:09:01.77,Default,,0000,0000,0000,,so there are two possible value of y
Dialogue: 0,0:09:01.77,0:09:04.04,Default,,0000,0000,0000,,I'm just trying to make my life easier
Dialogue: 0,0:09:04.04,0:09:09.61,Default,,0000,0000,0000,,so we have a joint probability \Ndistribution x_i and y_j
Dialogue: 0,0:09:09.61,0:09:12.05,Default,,0000,0000,0000,,this is our joint probability, as before
Dialogue: 0,0:09:12.05,0:09:14.88,Default,,0000,0000,0000,,and now we have a joint information
Dialogue: 0,0:09:14.88,0:09:18.38,Default,,0000,0000,0000,,which we shall call I of X and Y
Dialogue: 0,0:09:18.38,0:09:21.02,Default,,0000,0000,0000,,this is the information
Dialogue: 0,0:09:21.02,0:09:21.27,Default,,0000,0000,0000,,that's inherent in the joint set of events
Dialogue: 0,0:09:21.27,0:09:24.23,Default,,0000,0000,0000,,X and Y
Dialogue: 0,0:09:24.23,0:09:26.39,Default,,0000,0000,0000,,in our case, it being sunny and not sunny,\Nraining and not raining
Dialogue: 0,0:09:26.39,0:09:29.84,Default,,0000,0000,0000,,and this just takes the same form as before
Dialogue: 0,0:09:29.84,0:09:32.27,Default,,0000,0000,0000,,we sum over all different possibilities
Dialogue: 0,0:09:32.27,0:09:42.30,Default,,0000,0000,0000,,sunny-raining, not sunny-raining, \Nsunny-not raining, not sunny-not raining
Dialogue: 0,0:09:42.30,0:09:45.99,Default,,0000,0000,0000,,this is why one shouldn't try to enumerate these things
Dialogue: 0,0:09:45.99,0:09:53.14,Default,,0000,0000,0000,,p of x_i y_j logarithm of p of x_i y_j
Dialogue: 0,0:09:53.14,0:09:55.75,Default,,0000,0000,0000,,so this is the amount of information that's
Dialogue: 0,0:09:55.75,0:09:57.28,Default,,0000,0000,0000,,inherent with these two sets of events \Ntogether
Dialogue: 0,0:09:57.28,0:09:59.01,Default,,0000,0000,0000,,and of course, we still have this, if you like the
Dialogue: 0,0:10:00.80,0:10:03.89,Default,,0000,0000,0000,,marginal information, the information\Nof X on its own
Dialogue: 0,0:10:10.81,0:10:11.08,Default,,0000,0000,0000,,which is now just the sum over events x \Non its own
Dialogue: 0,0:10:13.47,0:10:13.77,Default,,0000,0000,0000,,of the marginal distribution
Dialogue: 0,0:10:13.77,0:10:14.84,Default,,0000,0000,0000,,why it's called "marginal" I don't know
Dialogue: 0,0:10:14.84,0:10:17.20,Default,,0000,0000,0000,,it's just the probability for X on its own
Dialogue: 0,0:10:22.08,0:10:22.71,Default,,0000,0000,0000,,p of X_i log base two of X_i
Dialogue: 0,0:10:22.71,0:10:24.83,Default,,0000,0000,0000,,and similarly we can talk about
Dialogue: 0,0:10:32.31,0:10:33.43,Default,,0000,0000,0000,,I of Y is minus the sum over j\Np of Y_j log to the base 2 of
Dialogue: 0,0:10:35.83,0:10:36.08,Default,,0000,0000,0000,,p of Y_j
Dialogue: 0,0:10:38.37,0:10:38.99,Default,,0000,0000,0000,,this is the amount of information
Dialogue: 0,0:10:40.97,0:10:41.22,Default,,0000,0000,0000,,inherent whether it's sunny or not sunny
Dialogue: 0,0:10:43.32,0:10:43.63,Default,,0000,0000,0000,,it could be up to a bit of information
Dialogue: 0,0:10:46.33,0:10:47.13,Default,,0000,0000,0000,,if it's probability one half of being \Nsunny or not sunny
Dialogue: 0,0:10:49.21,0:10:49.68,Default,,0000,0000,0000,,then there's a bit of information let me \Ntell you in Santa Fe
Dialogue: 0,0:10:51.68,0:10:52.22,Default,,0000,0000,0000,,there's far less than a bit of information
Dialogue: 0,0:10:52.22,0:10:53.37,Default,,0000,0000,0000,,on whether it's sunny or not
Dialogue: 0,0:10:53.37,0:10:54.72,Default,,0000,0000,0000,,because it's sunny most of the time
Dialogue: 0,0:10:54.72,0:10:56.40,Default,,0000,0000,0000,,similarly, raining or not raining
Dialogue: 0,0:10:56.53,0:10:58.44,Default,,0000,0000,0000,,could be up to a bit of information
Dialogue: 0,0:11:00.24,0:11:00.49,Default,,0000,0000,0000,,if each of these probabilities is 1/2
Dialogue: 0,0:11:02.86,0:11:03.64,Default,,0000,0000,0000,,again we're in the high desert here
Dialogue: 0,0:11:05.08,0:11:05.47,Default,,0000,0000,0000,,it's normally not raining
Dialogue: 0,0:11:05.47,0:11:07.85,Default,,0000,0000,0000,,so, you've far less than a bit of information
Dialogue: 0,0:11:09.83,0:11:10.43,Default,,0000,0000,0000,,on the question whether it's raining or not raining
Dialogue: 0,0:11:12.77,0:11:13.02,Default,,0000,0000,0000,,so, we have joint information
Dialogue: 0,0:11:14.89,0:11:16.82,Default,,0000,0000,0000,,constructed out of joint probabilities
Dialogue: 0,0:11:19.14,0:11:19.39,Default,,0000,0000,0000,,marginal information, or information on the original variables on their own,
Dialogue: 0,0:11:21.61,0:11:22.94,Default,,0000,0000,0000,,constructed\Nout of marginal probabilities
Dialogue: 0,0:11:26.13,0:11:26.48,Default,,0000,0000,0000,,and let me end this little section by defining
Dialogue: 0,0:11:28.64,0:11:32.48,Default,,0000,0000,0000,,a very useful quantity which is called the \Nmutual information
Dialogue: 0,0:11:38.81,0:11:42.43,Default,,0000,0000,0000,,the mutual information, which is defined to be
Dialogue: 0,0:11:42.43,0:11:46.65,Default,,0000,0000,0000,,I( X ...I normally define it with this little colon
Dialogue: 0,0:11:46.65,0:11:49.78,Default,,0000,0000,0000,,right in the middle, because it looks nice\Nand symmetrical
Dialogue: 0,0:11:49.78,0:11:54.96,Default,,0000,0000,0000,,and we'll see that this isn't symmetrical
Dialogue: 0,0:11:54.96,0:11:58.09,Default,,0000,0000,0000,,it's the information in X plus the information in Y
Dialogue: 0,0:11:58.09,0:12:01.21,Default,,0000,0000,0000,,minus the information in X and Y taken together
Dialogue: 0,0:12:01.21,0:12:03.96,Default,,0000,0000,0000,,it's possible to show that this is always greater or equal to zero
Dialogue: 0,0:12:03.96,0:12:09.11,Default,,0000,0000,0000,,and this mutual information can be thought of as the amount of information
Dialogue: 0,0:12:09.11,0:12:11.98,Default,,0000,0000,0000,,the variable X has about Y
Dialogue: 0,0:12:11.98,0:12:16.55,Default,,0000,0000,0000,,if X and Y are completely uncorrelated, so it's completely\Nuncorrelated whether it's sunny
Dialogue: 0,0:12:16.55,0:12:18.55,Default,,0000,0000,0000,,or not sunny or raining or not raining
Dialogue: 0,0:12:18.55,0:12:22.41,Default,,0000,0000,0000,,then this will be zero
Dialogue: 0,0:12:22.41,0:12:24.72,Default,,0000,0000,0000,,however, in the case of sunny and not sunny
Dialogue: 0,0:12:24.72,0:12:28.97,Default,,0000,0000,0000,,raning and not raining, they are very correlated
Dialogue: 0,0:12:28.97,0:12:32.17,Default,,0000,0000,0000,,in the sense that once you know that it's sunny
Dialogue: 0,0:12:32.17,0:12:33.57,Default,,0000,0000,0000,,it's probabiy not raining, even though
Dialogue: 0,0:12:33.57,0:12:35.37,Default,,0000,0000,0000,,sometimes that does happen here in Santa Fe
Dialogue: 0,0:12:35.37,0:12:36.39,Default,,0000,0000,0000,,and so in that case, you'd expect
Dialogue: 0,0:12:36.39,0:12:37.66,Default,,0000,0000,0000,,to find a large amount of mutual information
Dialogue: 0,0:12:37.66,0:12:39.80,Default,,0000,0000,0000,,in most places in fact, you'll find that knowing
Dialogue: 0,0:12:39.80,0:12:41.15,Default,,0000,0000,0000,,whether it's sunny or not sunny
Dialogue: 0,0:12:41.15,0:12:42.36,Default,,0000,0000,0000,,gives you a very good prediction
Dialogue: 0,0:12:42.36,0:12:45.13,Default,,0000,0000,0000,,about whether it's raining or it's not raining
Dialogue: 0,0:12:45.13,0:12:49.15,Default,,0000,0000,0000,,mutual information measures the amount of information\Nthat X can tell us about Y
Dialogue: 0,0:12:49.15,0:12:53.40,Default,,0000,0000,0000,,it's symmetric, so it tells us the amount of information that \NY can tell us about X
Dialogue: 0,0:12:53.40,0:12:55.91,Default,,0000,0000,0000,,and another way of thinking about it
Dialogue: 0,0:12:55.91,0:12:58.36,Default,,0000,0000,0000,,is that it's the amount of information
Dialogue: 0,0:12:58.36,0:13:00.04,Default,,0000,0000,0000,,that X and Y hold in common
Dialogue: 0,0:13:00.04,0:13:05.96,Default,,0000,0000,0000,,which is why it's called "mutual information"