Showing Revision 3 created 05/25/2016 by Udacity Robot.

1. Title:
   Downloading Enron Data - Intro to Machine Learning
2. Description:

3. 80 0:00 - 0:03
   Now that I've defined a person of interest it's time to get our
4. 2770 0:03 - 0:04
   hands dirty with the dataset.
5. 3840 0:04 - 0:07
   So here's the path that I followed to find the dataset.
6. 6910 0:07 - 0:09
   I started on Google as I always do.
7. 9370 0:09 - 0:11
   My Google for Enron emails.
8. 10760 0:11 - 0:13
   The first thing that pops up is the Enron Email Dataset.
9. 12860 0:13 - 0:16
   You can see that it's a dataset that's, that's very famous.
10. 16750 0:17 - 0:20
    Many have gone before us in studying this for many different purposes.
11. 20350 0:20 - 0:22
    It has it's own Wikipedia page.
12. 21980 0:22 - 0:25
    It has a really interesting article that I recommend you
13. 24630 0:25 - 0:28
    read from the MIT Technology Review about the many uses of
14. 28000 0:28 - 0:30
    this dataset throughout the years.
15. 29680 0:30 - 0:32
    But the first link is the dataset itself.
16. 32240 0:32 - 0:34
    Let's follow that link.
17. 33510 0:34 - 0:37
    This takes us to a page from the Carnegie Mellon CS department.
18. 37400 0:37 - 0:40
    It gives a little bit of background on the dataset and
19. 40030 0:40 - 0:41
    if we scroll down a little ways.
20. 42560 0:43 - 0:44
    We see this link right here.
21. 43700 0:44 - 0:46
    This is the actual link to the dataset.
22. 45990 0:46 - 0:48
    Below that is a little bit more information.
23. 48260 0:48 - 0:52
    If you click on this it's going to download a TGZ file.
24. 51960 0:52 - 0:55
    Which you can see I've already downloaded down here.
25. 54880 0:55 - 0:56
    If you do this on your own.
26. 55830 0:56 - 0:59
    It took me nearly half an hour to download the whole dataset.
27. 59050 0:59 - 1:02
    So I recommend that you start the download and
28. 61810 1:02 - 1:04
    then walk away to do something else.
29. 64030 1:04 - 1:06
    Once you have the data set you'll need to unzip it.
30. 66440 1:06 - 1:09
    So move it into the directory where you want to work and
31. 69340 1:09 - 1:11
    then you can run a command like this.
32. 71240 1:11 - 1:17
    There's no real magic here I just googled how to unzip .tgz and
33. 76820 1:17 - 1:18
    found a command like this.
34. 78170 1:18 - 1:21
    Again this will take a few minutes.
35. 80530 1:21 - 1:24
    When you're done with that you'll get a directory called enron mail.
36. 84170 1:24 - 1:26
    And then CD into maildir.
37. 86210 1:26 - 1:28
    Here's the data set.
38. 87870 1:28 - 1:33
    It's organized into a number of directories, each of which belongs to a person.
39. 92680 1:33 - 1:36
    You see that there's so many here I can't even fit them all on one page.
40. 96260 1:36 - 1:40
    In fact, you'll find that there's over 150 people in this dataset.
41. 99630 1:40 - 1:42
    Each on is identified by their last name and
42. 101530 1:42 - 1:43
    the first letter of their first name.
43. 103260 1:43 - 1:47
    So, looking through on a very superficial level, I see Jeff Skilling.
44. 106850 1:47 - 1:48
    Let's see if I can find Ken Lay.
45. 108110 1:48 - 1:50
    Looks like he might be up here.
46. 110450 1:50 - 1:52
    Yep, there's Ken Lay.
47. 112050 1:52 - 1:56
    Of course, a whole bunch of people I've never heard of.
48. 115730 1:56 - 1:57
    And remember, my question is,
49. 117270 1:57 - 2:00
    how many of the persons of interest do I have emails from?
50. 120330 2:00 - 2:03
    Do I have enough persons of interest, do I have their emails,
51. 123070 2:03 - 2:06
    that I could start describing the patterns in those emails,
52. 125680 2:06 - 2:09
    using supervised classification algorithms?
53. 128538 2:09 - 2:11
    And so the way that I answered this question was,
54. 130639 2:11 - 2:14
    again, using some work by hand basically.
55. 133980 2:14 - 2:18
    I took my list of persons of interest and for
56. 137710 2:18 - 2:22
    each person on that list I just looked for their name in this directory.
57. 141660 2:22 - 2:25
    Let's go back to that, remind ourselves what it looked like.
58. 144590 2:25 - 2:26
    You can see the annotated list here.
59. 146020 2:26 - 2:29
    You might have been wondering what these letters were before each of the names.
60. 148540 2:29 - 2:30
    These are my notes that I wrote to myself.
61. 150460 2:30 - 2:35
    As to whether I actually have the inbox of each of each of these people.
62. 154750 2:35 - 2:39
    So Ken Lay and Jeff Skilling we already found.
63. 158800 2:39 - 2:42
    But then it started to become a little more difficult.
64. 161594 2:42 - 2:46
    So you can see there are many, many people than I have n's next to their name.
65. 165630 2:46 - 2:51
    And that means no I don't have, for example, Scott Yeager.
66. 170750 2:51 - 2:57
    If I go over to the dataset, I don't see a Yeager down here.
67. 176850 2:57 - 2:59
    So Scott Yeager is a person who I'd love to have his inbox.
68. 179430 2:59 - 3:03
    I'd love to have some emails to in and from him, but I don't.
69. 183490 3:03 - 3:07
    As it turns out, I don't have the email inboxes of a lot of people.
70. 187400 3:07 - 3:08
    So I'll be honest,
71. 188070 3:08 - 3:11
    at this point I was actually really just discouraged about the possibility of
72. 190890 3:11 - 3:13
    using this as a project at all.
73. 193270 3:13 - 3:17
    I think I counted something like four or five people that I had their inboxes.
74. 196880 3:17 - 3:20
    And while that might be a few hundred emails or something like that.
75. 201160 3:21 - 3:25
    There's really no chance that with four examples of persons of interest I
76. 205210 3:25 - 3:30
    could start to describe the patterns of persons of interest as a whole.
77. 209760 3:30 - 3:33
    In the next video, though, I want to give you a key insight that I
78. 212830 3:33 - 3:36
    had that gave this project a second chance.
79. 216280 3:36 - 3:39
    A different way of trying to access the email inboxes of
80. 218900 3:39 - 3:40
    the persons of interest.