YouTube

Got a YouTube account?

New: enable viewer-created translations and captions on your YouTube channel!

English subtitles

← How we found the worst place to park in New York City — using big data | Ben Wellington | TEDxNewYork

Get Embed Code
12 Languages

Showing Revision 16 created 02/27/2015 by Ivana Korom.

  1. Six thousand miles of road,
  2. 600 miles of subway track,
  3. 400 miles of bike lanes,
  4. and a half a mile of tram track,
  5. if you've ever been to Roosevelt Island.
  6. These are the numbers that make up
    the infrastructure of NYC,
  7. these are the statistics
    of our infrastructure.
  8. They're the kind of numbers
    released in reports by city agencies.
  9. For example, the Department
    of Transportation will probably tell you
  10. how many miles of road they maintain.
  11. The MTA will boast how many miles
    of subway track there are.
  12. But most city agencies give us statistics.
  13. This is from a report this year
    from the Taxi & Limousine Commission,
  14. where we've learned that there is
    about 13,500 taxis here in NYC.
  15. Pretty interesting, right?
  16. But did you ever think about
    where these numbers came from?
  17. Because for these numbers to exist
    somebody at the city agency
  18. has to stop and say hmm, here's a number
    that somebody might want to know.
  19. Here's a number
    that our citizens want to know.
  20. So they go back to their raw data,
  21. they count, they add, they calculate,
  22. and then they put out reports.
  23. And those reports
    will have numbers like this.
  24. The problem is, how do they know
    all of our questions?
  25. We have lots of questions.
  26. In fact, in some ways there's literally
    an infinite number of questions
  27. that we can ask about our city.
  28. So the agencies can never keep up.
  29. So the paradigm isn't exactly working
  30. and I think our policy makers realize that
  31. because in 2012, Mayor Bloomberg
    signed into law what he called
  32. the most ambitious and comprehensive
    open data legislation in the country.
  33. In a lot of ways he's right.
  34. In the last two years the city's released
    1,000 data sets on our open data portal
  35. and, it's pretty awesome.
  36. You look at data like this,
  37. and instead of counting
    the number of cabs,
  38. we can start to ask different questions.
  39. So I had a question:
    When is rush hour in NYC?
  40. It can be pretty bothersome.
    When is rush hour exactly?
  41. And I thought to myself,
    these cabs aren't just numbers,
  42. these are GPS recorders driving around
    in our city's streets recording
  43. each and every right they take.
  44. There's data there.
  45. And I looked at that data
    and I made a plot
  46. of the average speed of taxis in NYC
    throughout the day.
  47. You can see that from around midnight
    to around 5:18 AM, speed increases,
  48. and at that point, things turn around.
  49. They get slower, slower and slower
    until about 8:35 AM
  50. when they end up at 11.5 mph.
  51. The average taxi is going at 11.5 mph
    in our city streets,
  52. and it turns out it stays that way
  53. for the entire day.
  54. (Laughter)
  55. So I said to myself, I guess
    there's no rush hour in NYC,
  56. there's just a "rush day."
  57. (Laughter)
  58. Makes sense.
  59. This is important
    for a couple of reasons.
  60. If you are a transportation planner,
    this might be pretty interesting to know.
  61. But if you want to get somewhere quickly
  62. you now know to set your alarm
    for 4:45 AM and you're all set.
  63. New York, right?
  64. But there's story behind this data,
  65. it wasn't just available as it turns out.
  66. It actually came from something called
    a Freedom of Information Law Request,
  67. or a FOIL Request.
  68. This is a form you can find on
    the Taxi & Limousine Commission website.
  69. In order to access this data,
    you need to go get this form,
  70. fill it out, and they will notify you.
  71. And a guy name Chris Whong
    did exactly that.
  72. Chris went down and they told him,
  73. "Just bring a brand new hard drive
    to our office,
  74. leave it here for 5 hours,
    we'll copy the data and you take it back."
  75. And that's where this data came from.
  76. Now, Chris is the kind of guy
    that wants to make the data public,
  77. so it ended up online for all to use
    and that's where this graph came from.
  78. And the fact that it exists is amazing.
  79. These GPS recorders - really cool!
  80. But the fact that we have citizens
    walking around with hard drives
  81. picking up data from city agencies
    to make it public -
  82. it was already kind of public,
    you could get to it,
  83. but it was "public", it wasn't public.
  84. And we can do better than that as a city,
  85. we don't need our citizens
    walking around with hard drives.
  86. Now, not every dataset
    is behind a FOIL request.
  87. Here's a map I made with
    the most dangerous intersections in NYC
  88. based on cyclist accidents.
  89. So the red areas are more dangerous.
  90. What it shows is first
    the East side of Manhattan,
  91. especially in the lower area of Manhattan,
    has more cycle accidents.
  92. That might makes sense
  93. because there are more cyclist
    coming off the bridges over there.
  94. But there's other hotspots worth studying.
  95. There's Williamsburg.
    There's Roosevelt Avenue in Queens.
  96. This is exactly the type of data
    we need for vision zero.
  97. This is exactly what we're looking for.
  98. But there's story
    behind this data as well.
  99. This data didn't just appear.
  100. How many of you guys know this logo?
  101. Yeah, I see some shakes.
  102. Have you ever tried to copy
    and paste data out of a PDF
  103. and make sense of it?
  104. I see more shakes.
  105. More of you tried to copying and pasting
    than knew the logo. I like that.
  106. What happen is, the data
    that you just saw was actually on a PDF.
  107. In fact, hundreds, and hundreds,
    of pages of PDF put out by our own NYPD,
  108. and in order to access it,
  109. you either have to copy and paste
    for hundred and hundred of hours,
  110. or you could be John Krauss.
  111. John Krauss is like,
  112. I'm not going to copy and paste this data,
    I'm going to write a program.
  113. It's called the NYPD Crash Data Band-Aid.
  114. And it goes to the NYPD's website
    and it would download PDFs.
  115. Every day with it would search;
  116. if it found a PDF, it would download it,
  117. and it would run
    some PDF-scraping program,
  118. and out would come the text
  119. and it would go on the Internet,
    and people could make maps like that.
  120. And the fact that the data is here,
    that we can have access to it -
  121. every accident, by the way, is a row
    on this table.
  122. You can imagine how many PDF that is.
  123. The fact that we
    have access to that is great.
  124. But let's not release it in PDF form.
  125. Because then we're having our citizens
    write PDF scrapers.
  126. It's not the best use
    of our citizens' time,
  127. and we, as a city,
    can do better than that.
  128. The good news is that
    the de Blasio Administration
  129. actually released this data
    a few months ago,
  130. so now, we can have access to it.
  131. But there's a lot of data
    still entombed in PDF.
  132. For example our crime data,
    still is only available in PDF.
  133. And not just our crime data,
  134. our own city budget.
  135. Our city budget is only
    readable right now in PDF form.
  136. And it's not just us
    that can't analyze it -
  137. our own legislators
    who vote for the budget,
  138. also only get it in PDF.
  139. So our legislators cannot analyze
    the budget that they are voting for.
  140. And I think as a city we can do
    a little better than that as well.
  141. Now, there's a lot of data
    that's not hidden in PDFs.
  142. This is an example of a map I made.
  143. And this is the dirtiest waterways in NYC.
  144. How do I measure dirty?
  145. Well, it's kind of a little weird,
  146. but I looked at the level
    of fecal coliform,
  147. which is a measurement of fecal matter
    in each of our waterways.
  148. The larger the circle,
    the dirtier the water.
  149. The large circles are dirty waters,
    the smaller circles are cleaner.
  150. What you see is inland waterways.
  151. This is all data that was sampled
    by the city over the last 5 years.
  152. And inland waterways are,
    in general, dirtier.
  153. That makes sense, right?
  154. And I learned a few things from this.
  155. Number 1: never swim in anything
    that ends in creek or canal.
  156. Number 2: I also found
    the dirtiest waterways in New York City
  157. by this measure, one measure.
  158. In Coney Island Creek,
  159. which is not Coney Island you swim in,
    luckily, it's on the other side.
  160. But Coney Island Creek, 94% of samples
    taken over the last 5 years
  161. have had fecal levels so high,
  162. that it would be against state law
    to swim in the water.
  163. And this is not the kind of fact
    that you're going to see
  164. boasted in a city report
    or on the front page of nyc.gov.
  165. You're not going to see it there,
  166. but the fact that we can
    get to that data, is awesome.
  167. Once again, it wasn't super easy,
  168. because this data was not
    on the open data portal.
  169. If you were to go to the open data portal,
  170. you'd see just a snippet of it,
    a year or a few months.
  171. It was actually on the Department
    of Environmental Protection's website.
  172. Each one of these links is an Excel sheet,
    and this Excel sheet is different.
  173. Every heading is different:
    you copy, paste, reorganize.
  174. When you do you can make maps
    and that's great, but once again,
  175. we can do better than that as a city,
    we can normalize things.
  176. We're getting there because
    there's this website that Socrata makes,
  177. called the Open Data Portal NYC.
  178. This is where 1100 data sets,
    that don't suffer
  179. from the things I told you live,
  180. and that number is growing,
    and that's great.
  181. You can download data in any format,
    be it CSV or PDF or Excel document.
  182. Whatever you want,
    you can download the data that way.
  183. The problem is, once you do,
  184. you'll find that each agency
    codes their addresses differently.
  185. So, one is street name,
    intersection street,
  186. street, borough, address building,
    building, address.
  187. So, once again, you're spending time,
    even when we have this portal,
  188. you're spending time
    normalizing our address field.
  189. I think that's not the best use
    of our citizens' time,
  190. we can do better than that as a city.
  191. We can standardize our addresses.
  192. If we do, we can get more maps like this.
  193. This is a map of fire hydrants
    in New York City.
  194. But not just any fire hydrant.
  195. These are the top 250
    grossing fire hydrants
  196. in terms of parking tickets.
  197. (Laughter)
  198. So I learned a few things from this map.
  199. Number 1: just don't park
    on the Upper East side.
  200. Just don't. No matter where you park,
    you will get a hydrant ticket.
  201. Number 2: I found the two highest
    grossing hydrants in all of New York City.
  202. They are on the Lower East side,
  203. and they are bringing in over
    55,000 dollars a year in parking tickets.
  204. And that seemed a little strange to me
    when I noticed it,
  205. so I did a little digging,
    and it turns out
  206. what you had is a hydrant
    and something called a curb extension,
  207. which is like a seven-foot space
    to walk on,
  208. and then a parking spot.
  209. So these cars came along and the hydrant -
  210. "It's all the way over there, I'm fine,"
  211. and there was actually a parking spot
    painted there beautifully for them.
  212. They would park there and the NYPD
    disagree with the designation,
  213. and would ticket them.
  214. And it wasn't just me
    who found a parking ticket.
  215. This is the Google street view car
    driving by, finding same parking ticket.
  216. So I wrote about this
    on my blog, on I Quant NY,
  217. and the DOT responded and they said,
  218. "While the DOT has not received
    any complaints about this location,
  219. we will review the roadway markings
    and make any appropriate alterations."
  220. I thought to myself, you know,
    typical government response,
  221. all right, moved on with my life.
  222. But then, a few weeks later,
    something incredible happened.
  223. They repainted the spot.
  224. And for a second I thought
    I saw the future of open data
  225. because think about what happened here.
  226. For five years, this spot
    was being ticketed, and it was confusing.
  227. And then a citizen found something,
    they told the city and within a few weeks,
  228. the problem was fixed. It's amazing.
  229. A lot of people see open data
    as being a watch dog, it's not.
  230. It's about being a partner.
  231. We can empower our citizens to be
    better partners for government,
  232. and it's not that hard.
  233. All we need are a few changes.
  234. If you're FOILing data,
  235. if you seeing your data
    being FOILed over and over again,
  236. let's release it to the public, that's
    a sign that it should be made public.
  237. And if you're a government agency
    releasing a PDF,
  238. let's pass a legislation that requires you
    to post it with your underlying data,
  239. because that data
    is coming from somewhere.
  240. I don't know where,
    but you can release it with the PDF.
  241. And let's adopt and share
    some open data standards.
  242. Let's start with our addresses
    here in New York City.
  243. Let's just start
    normalizing our addresses.
  244. Because New York is a leader in open data.
  245. Despite all this, we're absolutely
    a leader in open data,
  246. and if we start normalizing things,
    and set an open data standard,
  247. others will follow.
  248. The state will follow,
    maybe the federal government,
  249. other countries could follow,
  250. and we're not that far off from a time
    where you can write one program
  251. and map information from a 100 countries.
  252. It's not science fiction,
    we're actually quite close.
  253. And by the way, who are we
    empowering with this?
  254. Because it's not just John Krauss,
    it's not just Chris Whong.
  255. There are hundred of meetups
    going around in New York City right now,
  256. active meetups.
  257. There are thousands of people
    attending these meetups.
  258. These people are going after work
    and on weekends,
  259. and they're attending these meetups
    to look at open data,
  260. and make our city a better place.
  261. Groups like BetaNYC who just last week,
    released something called citygram.nyc
  262. that allows you to subscribe
    to 311 complaints
  263. around your own home,
    or around your office.
  264. You put in your address,
    you get local complaints.
  265. And it's not just the tech community
    that are after these things.
  266. It's urban planners like the students
    I teach at Pratt.
  267. It's policy advocates, it's everyone,
  268. it's citizens from a diverse
    set of backgrounds.
  269. And with some small incremental changes,
  270. we can unlock the passion
    and the ability of our citizens
  271. to harness open data
    and make our city even better,
  272. whether is one data set
    or one parking spot at a time.
  273. Thank you.
  274. (Applause)