we begin with a problem
Alice and Bob leave in tree forts
which are far apart with no line of sight between them
and they need to communicate
so they decide to run a wire between the two houses
they pull a wire tight
and attach it tin can to each end
allowing them to send their voices faintly along the wire
however there is a problem
noise
whenever there is a high wind
it becomes impossible to hear the signal over the noise
so they need a way to increase the energy level of the signal
to separate it from the noise
this gives bob an idea
they can simply pluck the wire
which is much easier to detect over the noise
but this leads to a new problem
how do they encode their messages as plucks?
well since they want to play board games across a distance
they tackle the most common messages first
the outcome of two dice rolls
in this case the messages they are sending
can be thought of as a selection from a finite number of symbols
in this case the 11 possible numbers
which we call a discrete source
at first they decide to use the simplest method
they send the result as the number of plucks
so to send a 3 they send three plucks
9 is nine plucks and 12 is twelve plucks
however they soon realize that this takes much longer than it needs to
from practice they find that their maximum plucks speed is
two plucks per second
any faster and they will get confused
so two plucks per second can be thought of as the rate
or capacity for sending information in this way
and it turns out that the most common roll is a 7
so it takes 3.5 seconds to send the number seven
Alice then realizes they can do much better
if they change their coding strategy
she realizes that the odds of each number being sent follow a simple pattern
there is one way to roll a 2, two ways to roll a 3
three ways to roll a 4, four ways to roll a 5
five ways to roll a 6 and six ways to roll a 7, the most common
and five ways to roll an 8
four ways for a 9 and so on
back to one way for a twelve
and this is the graph showing the number of ways each result can occur
and the pattern is obvious
so now let's change the graph to number of plucks versus each symbol
she proceeds by mapping the most common number, seven
to the shortest signal one pluck
she then proceeds to the next most probable number
and if there is a tie she picks one at random
in this case she selects six to be two plucks
and then eight to be three plucks
and then back to five to be four plucks
and nine as five plucks
and back and forth until we reach 12
which is assigned to 11 plucks
now the most common number seven
can be sent in less than a second
a huge improvement
this symbol change allows them to send more information
in the same amount of time on average
in fact this coding strategy is optimal for this simple example
in that it impossible for you to come up with a shorter method
of sending two dice rolls using identical plucks
however after playing with the wire for some time
Bob hit on a new idea