Got a YouTube account?

New: enable viewer-created translations and captions on your YouTube channel!

English (United States) subtitles

← Best Practices for Scraping - Data Wranging with MongoDB

Get Embed Code
5 Languages

Showing Revision 1 created 09/26/2016 by Udacity Robot.

  1. So I mentioned earlier, it's this kind of thing that can really make
  2. you want to tear your hair out. As you can see, I've been in
  3. lots of these situations. Okay, so we're facing a challenging situation, in that
  4. we've got to do some scraping from a site that's really making us work
  5. for it. Let's talk about a best practice for screen scraping. So the
  6. first thing we want to do here is as I mentioned is, look
  7. at how the browser itself makes requests. And with the developer tools, we
  8. have quite a few ways in which we can do that. We can also
  9. look at the wire traffic if we really needed to
  10. using something like Wireshark. Then we want to emulate that
  11. in our code. Now if everything blows up, we're going to
  12. need to take a look at our HTTP traffic in some
  13. way. And then just return to one and work through
  14. this again until we get it right. If we follow this
  15. process, we have a pretty effective strategy, for dealing with any
  16. problems we have making requests from sites we'd like to scrape.