Showing Revision 6 created 05/24/2016 by Udacity Robot.

1. Title:
   Finishing the Web Crawler - Intro to Computer Science
2. Description:

3. 0 0:00 - 0:04
   So let's remember the code we had at the end of unit 2 for crawling the web.
4. 4000 0:04 - 0:09
   So we used 2 variables. We initialized "tocrawl" to the seed, a list containing just the seed,
5. 9000 0:09 - 0:12
   and we're going to use "tocrawl" to keep track of the pages to crawl.
6. 12000 0:12 - 0:19
   We initialized "crawled" to the empty list, and we're keeping track of the pages we found using "crawled."
7. 19000 0:19 - 0:23
   Then we had a loop that would continue as long as there were pages left to crawl.
8. 23000 0:23 - 0:25
   We'd pop the last page off the "tocrawl" list.
9. 25000 0:25 - 0:32
   If it's not already crawled, then we'll union into "tocrawl" all the links that we can find on that page,
10. 32000 0:32 - 0:36
    and then we'll add that page to the list of pages we've already crawled.
11. 36000 0:36 - 0:41
    So now we want to figure out how to change this so instead of just finding all the URLs,
12. 41000 0:41 - 0:42
    we're building up our index.
13. 42000 0:42 - 0:46
    We're looking at the actual content of the pages, and we're adding it to our index.
14. 46000 0:46 - 0:52
    So the first change to make, we're updating the index, and we're going to change the return result.
15. 52000 0:52 - 0:56
    So instead of returning "crawled" what we want to return at the end is the index.
16. 56000 0:56 - 1:02
    `If we wanted to keep track of all the URLs crawled, we could still return "crawled" and return both "crawled" and "index,"
17. 62000 1:02 - 1:05
    but let's keep things simple and just return "index."
18. 65000 1:05 - 1:09
    That's what we really want for being able to respond to search queries.
19. 69000 1:09 - 1:12
    So now we have one other change to make, and this is the important one.
20. 72000 1:12 - 1:19
    We need to find a way to update the index to reflect all the words that are found on the page that we've just crawled.
21. 79000 1:19 - 1:22
    I'm going to make one change before we do that.
22. 82000 1:22 - 1:28
    Since both "getalllinks" and what we need to do to add the words to the index depend on the page,
23. 88000 1:28 - 1:33
    let's introduce a new variable and store the content of the page in that variable.
24. 93000 1:33 - 1:37
    This will save us from having to call "get_page" twice. "Get_page" is fairly expensive.
25. 97000 1:37 - 1:40
    It requires a web request to get the content of the page.
26. 100000 1:40 - 1:45
    It makes a lot more sense to store that in a new variable, and that will simplify this code.
27. 105000 1:45 - 1:48
    So now we just need to pass in content.
28. 108000 1:48 - 1:51
    So we have one missing statement,
29. 111000 1:51 - 1:56
    and I'll leave it to you to see if you can figure out statement we need there to finish the web crawler.
30. 116000 1:56 - 1:59
    When it's done, the result of "crawl-web," what we return as "index"
31. 119000 1:59 - 2:02
    should be an index of all the content we find starting from the seed.