Showing Revision 1 created 11/28/2012 by Amara Bot.

1. Title:
   10-08 Agents Of Reinforcement Learning
2. Description:

   Unit 10 08 Agents of Reinforcement Learning.mp4

3. 0 0:00 - 0:03
   Now here's where reinforcement learning comes into play:
4. 3000 0:03 - 0:06
   What if you don't know R--the Reward function?
5. 6000 0:06 - 0:09
   What if you don't even know P--the transition model of the world?
6. 9000 0:09 - 0:12
   Then you can't solve the Markov Decision Process
7. 12000 0:12 - 0:14
   because you don't have what you need to solve it.
8. 14000 0:14 - 0:16
   However, with reinforcement learning,
9. 16000 0:16 - 0:19
   you can learn R and P by interacting with the world
10. 19000 0:19 - 0:22
    or you can learn substitutes that will tell you
11. 22000 0:22 - 0:26
    as much as you know, so that you never actually have to compute with R and P.
12. 26000 0:26 - 0:30
    What you learn, exactly, depends on what you already know and what you want to do.
13. 30000 0:30 - 0:32
    So we have several choices.
14. 32000 0:32 - 0:36
    One choice is we can build a utility-based agent.
15. 36000 0:36 - 0:41
    So we're going to list agent types, based on what we know,
16. 41000 0:41 - 0:43
    what we want to learn,
17. 43000 0:43 - 0:45
    and what we then use once we've learned.
18. 45000 0:45 - 0:47
    So for a utility-based agent,
19. 47000 0:47 - 0:51
    if we already know T, the transition model,
20. 51000 0:51 - 0:54
    but we don't know R, the Reward model,
21. 54000 0:54 - 0:57
    then we can learn R--and use that,
22. 57000 0:57 - 1:01
    along with P, to learn our utility function;
23. 61000 1:01 - 1:04
    and then go ahead and use the utility function
24. 64000 1:04 - 1:07
    just as we did in normal Markov Decision Processes.
25. 67000 1:07 - 1:09
    So that's one agent design.
26. 69000 1:09 - 1:11
    Another design that we'll see in this Unit
27. 71000 1:11 - 1:14
    is called a Q-learning agent.
28. 74000 1:14 - 1:17
    In this one, we don't have to know P or R;
29. 77000 1:17 - 1:22
    and we learn a value function, which is usually denoted by Q.
30. 82000 1:22 - 1:26
    And that's a type of utility
31. 86000 1:26 - 1:28
    but, rather than being a utility over states,
32. 88000 1:28 - 1:32
    it's a utility of state action pairs--and that tells us:
33. 92000 1:32 - 1:36
    For any given state and any given action,
34. 96000 1:36 - 1:38
    what's the utility of that result--
35. 98000 1:38 - 1:42
    without knowing the utilities and rewards, individually?
36. 102000 1:42 - 1:45
    And then we can just use that Q directly.
37. 105000 1:45 - 1:49
    So we don't actually have to ever learn the transition model, P,
38. 109000 1:49 - 1:51
    with a Q-learning agent.
39. 111000 1:51 - 1:54
    And finally, we can have a reflex agent
40. 114000 1:54 - 1:57
    where, again, we don't need to know P and R to begin with;
41. 117000 1:57 - 2:02
    and we learn directly, the policy, pi of S;
42. 122000 2:02 - 2:05
    and then we just go ahead and apply pi.
43. 125000 2:05 - 2:09
    So it's called a reflex agent because it's pure stimulus response:
44. 129000 2:09 - 2:11
    I'm in a certain state, I take a certain action.
45. 131000 2:11 - 2:15
    I don't have to think about modeling the world, in terms of:
46. 135000 2:15 - 2:17
    What are the transitions--where am I going to go next?
47. 137000 2:17 -
    I just go ahead and take that action.