1 00:00:00,100 --> 00:00:02,050 ♪ (música) ♪ 2 00:00:03,620 --> 00:00:05,700 - [Narradora] Bienvenidos a Nobel Conversations. 3 00:00:07,000 --> 00:00:08,521 En este episodio, Josh Angrist y Guido Imbens 4 00:00:08,521 --> 00:00:09,282 están con Isaiah Andrews para discutir y discrepar 5 00:00:09,282 --> 00:00:09,662 sobre el papel del aprendizaje automático 6 00:00:09,662 --> 00:00:10,043 en la econometría aplicada. 7 00:00:10,043 --> 00:00:11,859 - [Isaiah] Entonces, por supuesto que hay muchos temas 8 00:00:11,859 --> 00:00:12,767 en los que ustedes están muy de acuerdo, pero me gustaría pasar a uno 9 00:00:12,767 --> 00:00:13,675 en el que tal vez tengan algunas discrepancias. 10 00:00:13,675 --> 00:00:15,127 Me gustaría escuchar algunas de sus opiniones 11 00:00:15,127 --> 00:00:15,853 sobre el aprendizaje automático 12 00:00:15,853 --> 00:00:16,580 y el papel que está desempeñando y va a desempeñar en la economía. 13 00:00:17,897 --> 00:00:18,833 - [Guido] He consultado algunos datos como el propietario----------------------. 14 00:00:18,833 --> 00:00:19,301 Vemos que no hay ningún documento publicado allí. 15 00:00:19,301 --> 00:00:19,535 Se hizo un experimento sobre algún algoritmo de búsqueda 16 00:00:19,535 --> 00:00:19,652 y la cuestión era... 17 00:00:19,652 --> 00:00:19,710 se trataba de clasificar cosas y cambiar la clasificación. 18 00:00:19,710 --> 00:00:19,739 Y estaba más o menos claro que iba a haber mucha heterogeneidad. 19 00:00:19,739 --> 00:00:19,754 Si buscas, digamos, una foto de Britney Spears -- 20 00:00:19,754 --> 00:00:19,761 realmente no importa dónde la clasifiques----------------------------------- porque vas a encontrar lo que estás buscando, 21 00:00:19,761 --> 00:00:19,765 ya sea que la pongas en la primera, segunda o tercera posición del ranking. 22 00:00:19,765 --> 00:00:19,767 Pero si estás buscando el mejor libro de econometría, 23 00:00:19,767 --> 00:00:19,768 si pones tu libro en primer lugar o en el décimo, 24 00:00:19,768 --> 00:00:19,768 eso va a suponer una gran diferencia 25 00:00:19,768 --> 00:00:19,768 en la frecuencia con la que la gente va a hacer clic en él. 26 00:00:19,768 --> 00:00:19,769 Así que ahí... 27 00:00:22,313 --> 00:00:23,276 [Josh] ¿Por qué necesito el aprendizaje automático para descubrir eso? 28 00:00:23,276 --> 00:00:24,240 Parece que --porque puedo descubrirlo simplemente.----------------------- 29 00:00:24,240 --> 00:00:24,984 - [Guido] En general -- 30 00:00:24,984 --> 00:00:25,356 - [Josh] Había un montón de posibles... 31 00:00:25,356 --> 00:00:25,542 - Quieres pensar en que hay un montón de características de los artículos, 32 00:00:25,542 --> 00:00:25,635 que quieres entender lo que impulsa la heterogeneidad en el efecto de -- 33 00:00:25,635 --> 00:00:25,681 - Pero solo estás prediciendo. 34 00:00:25,681 --> 00:00:25,704 En cierto sentido, estás resolviendo un problema de marketing. 35 00:00:25,704 --> 00:00:25,716 - No, es un efecto causal, 36 00:00:25,716 --> 00:00:25,722 - Es causal, pero no tiene contenido científico. 37 00:00:25,722 --> 00:00:25,725 Piensa en... 38 00:00:25,725 --> 00:00:25,726 - No, pero hay cosas similares en el ámbito médico. 39 00:00:25,726 --> 00:00:25,727 Si haces un experimento, 40 00:00:25,727 --> 00:00:25,727 puedes estar muy interesado en si el tratamiento funciona 41 00:00:25,727 --> 00:00:25,727 para algunos grupos o no. 42 00:00:25,727 --> 00:00:25,727 Y tienes un montón de características individuales, 43 00:00:25,727 --> 00:00:25,727 y quieres buscar sistemáticamente... 44 00:00:25,727 --> 00:00:25,727 - Sí. Soy escéptico sobre esa... 45 00:00:25,727 --> 00:00:25,727 esa especie de idea de que hay un efecto causal personal 46 00:00:25,727 --> 00:00:25,727 que me debería importar 47 00:00:25,727 --> 00:00:25,727 y que el aprendizaje automático puede descubrirlo de alguna manera que sea útil. 48 00:00:25,727 --> 00:00:25,727 Así que piensa en -- 49 00:00:25,727 --> 00:00:25,727 he hecho un montón de trabajo en las escuelas, 50 00:00:25,727 --> 00:00:25,727 ir a, digamos, una escuela charter,------------------------------------------------- 51 00:00:25,727 --> 00:00:25,727 una escuela privada financiada con fondos públicos, 52 00:00:25,727 --> 00:00:25,727 efectivamente, que es libre de estructurar su propio plan de estudios para el contexto allí. 53 00:00:25,727 --> 00:00:25,727 Algunos tipos de escuelas chárter generan espectaculares ganancias de rendimiento 54 00:00:25,727 --> 00:00:25,727 y en el conjunto de datos que produce ese resultado, 55 00:00:25,727 --> 00:00:25,727 tengo un montón de covariables. 56 00:00:25,727 --> 00:00:25,727 Así que tengo las puntuaciones de referencia 57 00:00:25,727 --> 00:00:25,727 y tengo los antecedentes familiares, 58 00:00:25,727 --> 00:00:25,727 la educación de los padres, el sexo del niño, la raza del niño. 59 00:00:25,727 --> 00:00:25,727 Y, bueno, en cuanto reúno media docena de ellas, 60 00:00:25,727 --> 00:00:25,727 tengo un espacio de muy alta dimensión. 61 00:00:25,727 --> 00:00:25,727 Definitivamente me interesan las características del curso 62 00:00:25,727 --> 00:00:25,727 de ese efecto del tratamiento, 63 00:00:25,727 --> 00:00:25,727 como por ejemplo si es mejor para las personas que provienen de familias con menores ingresos. 64 00:00:25,727 --> 00:00:25,727 Me cuesta creer que haya una aplicación para la versión de muy alta dimensión de eso, 65 00:00:25,727 --> 00:00:25,727 en la que descubrí que para los niños no blancos 66 00:00:25,727 --> 00:00:25,727 que tienen ingresos familiares altos 67 00:00:25,727 --> 00:00:25,727 pero puntuaciones de referencia en el tercer cuartil 68 00:00:25,727 --> 00:00:25,728 y que solo fueron a la escuela pública en el tercer grado pero no en el sexto. 69 00:00:25,728 --> 00:00:26,305 Así que eso es lo que produce ese análisis de alta dimensión. 70 00:00:26,305 --> 00:00:26,883 Es una declaración condicional muy elaborada. 71 00:00:30,200 --> 00:00:33,352 Hay dos cosas que están mal, en mi opinión. 72 00:00:33,352 --> 00:00:34,251 En primer lugar, no lo veo como... 73 00:00:34,251 --> 00:00:34,700 no puedo imaginar por qué es procesable. 74 00:00:34,700 --> 00:00:34,925 No sé por qué querrías actuar sobre ello. 75 00:00:34,925 --> 00:00:35,037 Y también sé que hay algún modelo alternativo que encaja casi igual, que lo invierte todo. 76 00:00:35,037 --> 00:00:35,093 Porque el aprendizaje automático 77 00:00:35,093 --> 00:00:35,121 no me dice que este es realmente el predictor que importa -- 78 00:00:35,121 --> 00:00:35,150 sólo me dice que este es un buen predictor. 79 00:00:36,122 --> 00:00:37,140 Así que creo que hay algo diferente en el contexto de las ciencias sociales. 80 00:00:37,140 --> 00:00:37,649 - [Guido] Creo que las aplicaciones de las ciencias sociales 81 00:00:37,649 --> 00:00:37,904 de las que hablas son aquellas en las que, creo, 82 00:00:37,904 --> 00:00:38,031 no hay una gran cantidad de heterogeneidad en los efectos. 83 00:00:38,031 --> 00:00:38,095 - [Josh] Bueno, podría haberla si me permites llenar ese espacio. 84 00:00:38,095 --> 00:00:38,127 - No... ni siquiera entonces. 85 00:00:38,127 --> 00:00:38,143 Creo que para muchas de esas intervenciones, 86 00:00:38,143 --> 00:00:38,151 se espera que el efecto sea del mismo signo para todos. 87 00:00:38,151 --> 00:00:38,155 Puede haber pequeñas diferencias en la magnitud, pero no es... 88 00:00:38,155 --> 00:00:38,157 Para muchas de estas defensas educativas - son buenas para todos. 89 00:00:38,157 --> 00:00:38,158 No es que sean malas para algunas personas y buenas para otras 90 00:00:38,158 --> 00:00:38,159 y eso es una especie ------------------------------------------------------------------------------------------------------------------------------------. 91 00:00:38,159 --> 00:00:38,829 Pero puede haber algo de variación en la magnitud, 92 00:00:38,829 --> 00:00:39,164 pero se necesitarían conjuntos de datos muy, muy grandes para encontrarlos. 93 00:00:39,164 --> 00:00:39,332 Estoy de acuerdo en que, en esos casos, probablemente no serían muy procesables de todos modos. 94 00:00:39,332 --> 00:00:39,416 Pero creo que hay muchos otros escenarios donde hay mucha más heterogeneidad. 95 00:00:39,416 --> 00:00:39,458 - Bueno, estoy abierto a esa posibilidad 96 00:00:39,458 --> 00:00:39,479 y creo que el ejemplo que has dado es esencialmente un ejemplo de marketing. 97 00:00:39,479 --> 00:00:39,489 - No, esos tienen implicaciones para ello y esa es la organización, 98 00:00:39,489 --> 00:00:39,494 si tienes que preocuparte por la... 99 00:00:39,494 --> 00:00:39,497 - Bueno, necesito ver ese documento. 100 00:00:39,497 --> 00:00:39,498 - Así que, la sensación que tengo es que... 101 00:00:39,498 --> 00:00:39,499 - Todavía no estamos de acuerdo en algo. 102 00:00:39,499 --> 00:00:39,499 - Sí. 103 00:00:39,499 --> 00:00:39,499 - No hemos convergido en todo. 104 00:00:39,499 --> 00:00:39,499 - Tengo esa sensación. [risas] 105 00:00:39,499 --> 00:00:39,499 - En realidad, hemos divergido en esto porque esto no estaba para discutir. 106 00:00:39,499 --> 00:00:39,499 [risas] 107 00:00:39,499 --> 00:00:39,499 - ¿Se está calentando un poco aquí? 108 00:00:39,499 --> 00:00:39,499 - ------------------------------------------------------es bueno. 109 00:00:39,499 --> 00:00:39,499 La sensación que tengo es, Josh, 110 00:00:39,499 --> 00:00:39,499 que no estás diciendo que estás seguro de que no hay manera de que haya una aplicación en la que estas cosas sean útiles. 111 00:00:39,499 --> 00:00:39,499 Estás diciendo que no estás convencido 112 00:00:39,499 --> 00:00:39,499 de las aplicaciones existentes hasta la fecha. 113 00:00:39,499 --> 00:00:39,499 - Es justo. 114 00:00:39,499 --> 00:00:39,499 - Estoy muy seguro. 115 00:00:39,499 --> 00:00:39,499 [risas] 116 00:00:39,499 --> 00:00:39,499 - En este caso. 117 00:00:39,499 --> 00:00:39,499 - Creo que Josh tiene un punto que incluso en los casos de predicción donde muchos de los métodos de aprendizaje de la máquina realmente brillan es donde hay un montón de heterogeneidad. 118 00:00:39,499 --> 00:00:39,499 - No te importan mucho los detalles, ¿verdad? 119 00:00:39,499 --> 00:00:39,499 - [Guido] Sí. 120 00:00:39,499 --> 00:00:39,499 - No tiene un ángulo de política o algo así. 121 00:00:39,499 --> 00:00:39,500 122 00:00:39,700 --> 00:00:41,327 123 00:00:42,901 --> 00:00:45,600 124 00:00:45,900 --> 00:00:47,290 125 00:00:48,400 --> 00:00:50,610 126 00:00:52,161 --> 00:00:56,282 127 00:00:57,831 --> 00:01:00,617 128 00:01:00,617 --> 00:01:02,493 129 00:01:02,493 --> 00:01:05,500 130 00:01:06,200 --> 00:01:07,867 131 00:01:07,867 --> 00:01:09,920 132 00:01:10,100 --> 00:01:12,500 133 00:01:13,300 --> 00:01:16,430 134 00:01:16,430 --> 00:01:18,100 135 00:01:18,600 --> 00:01:20,979 136 00:01:21,829 --> 00:01:23,417 137 00:01:23,417 --> 00:01:27,119 138 00:01:27,119 --> 00:01:29,195 139 00:01:29,195 --> 00:01:30,435 140 00:01:30,435 --> 00:01:32,100 141 00:01:32,100 --> 00:01:35,045 142 00:01:35,490 --> 00:01:37,280 143 00:01:37,610 --> 00:01:41,682 144 00:01:42,177 --> 00:01:43,427 145 00:01:43,427 --> 00:01:45,008 146 00:01:45,008 --> 00:01:47,665 147 00:01:47,665 --> 00:01:49,381 148 00:01:49,381 --> 00:01:51,911 149 00:01:51,911 --> 00:01:53,141 150 00:01:53,657 --> 00:01:57,300 151 00:01:58,000 --> 00:02:01,300 152 00:02:01,300 --> 00:02:03,900 153 00:02:03,900 --> 00:02:06,143 154 00:02:06,143 --> 00:02:08,000 155 00:02:08,000 --> 00:02:09,500 156 00:02:09,500 --> 00:02:12,603 157 00:02:12,603 --> 00:02:14,000 158 00:02:14,000 --> 00:02:15,740 159 00:02:15,740 --> 00:02:17,259 160 00:02:17,259 --> 00:02:20,045 161 00:02:20,045 --> 00:02:22,336 162 00:02:22,336 --> 00:02:24,428 163 00:02:25,225 --> 00:02:27,392 164 00:02:27,392 --> 00:02:29,399 165 00:02:29,399 --> 00:02:31,369 166 00:02:31,369 --> 00:02:33,703 167 00:02:33,703 --> 00:02:36,400 168 00:02:36,400 --> 00:02:37,800 169 00:02:37,800 --> 00:02:41,353 170 00:02:41,353 --> 00:02:43,207 171 00:02:43,576 --> 00:02:45,800 172 00:02:45,800 --> 00:02:49,795 173 00:02:49,795 --> 00:02:51,900 174 00:02:52,300 --> 00:02:55,199 175 00:02:55,199 --> 00:02:56,457 176 00:02:56,457 --> 00:02:58,741 177 00:02:58,741 --> 00:03:02,046 178 00:03:02,600 --> 00:03:05,760 179 00:03:07,273 --> 00:03:09,872 180 00:03:09,872 --> 00:03:12,406 181 00:03:12,406 --> 00:03:14,971 182 00:03:14,971 --> 00:03:17,800 183 00:03:18,166 --> 00:03:21,785 184 00:03:21,785 --> 00:03:23,000 185 00:03:23,000 --> 00:03:25,796 186 00:03:25,800 --> 00:03:28,100 187 00:03:28,300 --> 00:03:30,605 188 00:03:30,605 --> 00:03:31,797 189 00:03:31,797 --> 00:03:34,000 190 00:03:34,600 --> 00:03:36,600 191 00:03:36,600 --> 00:03:39,455 192 00:03:39,455 --> 00:03:41,200 193 00:03:41,800 --> 00:03:43,398 194 00:03:43,398 --> 00:03:45,350 195 00:03:45,350 --> 00:03:48,582 196 00:03:48,582 --> 00:03:50,965 197 00:03:51,486 --> 00:03:54,870 198 00:03:54,870 --> 00:03:57,586 199 00:03:57,940 --> 00:04:00,186 200 00:04:00,186 --> 00:04:01,633 201 00:04:01,633 --> 00:04:02,735 202 00:04:03,400 --> 00:04:08,100 203 00:04:09,777 --> 00:04:11,544 204 00:04:11,544 --> 00:04:13,466 205 00:04:13,466 --> 00:04:15,740 206 00:04:15,740 --> 00:04:18,614 207 00:04:18,614 --> 00:04:22,765 208 00:04:24,367 --> 00:04:27,600 209 00:04:28,200 --> 00:04:30,232 210 00:04:30,232 --> 00:04:31,700 211 00:04:34,169 --> 00:04:36,034 212 00:04:36,034 --> 00:04:37,600 213 00:04:37,600 --> 00:04:39,100 214 00:04:39,100 --> 00:04:40,900 215 00:04:40,900 --> 00:04:44,011 216 00:04:44,011 --> 00:04:46,955 217 00:04:47,906 --> 00:04:49,078 218 00:04:49,078 --> 00:04:51,400 219 00:04:51,700 --> 00:04:53,800 220 00:04:54,100 --> 00:04:56,600 221 00:04:57,250 --> 00:04:59,102 222 00:04:59,102 --> 00:05:04,918 223 00:05:06,315 --> 00:05:09,656 224 00:05:09,656 --> 00:05:11,069 225 00:05:12,252 --> 00:05:14,330 226 00:05:15,469 --> 00:05:17,900 227 00:05:18,400 --> 00:05:21,072 228 00:05:21,467 --> 00:05:23,996 229 00:05:23,996 --> 00:05:25,440 230 00:05:25,440 --> 00:05:27,200 231 00:05:27,200 --> 00:05:28,679 232 00:05:28,679 --> 00:05:30,833 233 00:05:30,833 --> 00:05:32,334 234 00:05:33,057 --> 00:05:34,771 235 00:05:35,820 --> 00:05:38,147 236 00:05:38,147 --> 00:05:41,187 237 00:05:41,187 --> 00:05:43,119 238 00:05:43,119 --> 00:05:45,347 239 00:05:45,347 --> 00:05:47,028 240 00:05:47,028 --> 00:05:49,487 241 00:05:49,917 --> 00:05:52,022 242 00:05:52,022 --> 00:05:53,704 243 00:05:54,156 --> 00:05:55,189 244 00:05:55,189 --> 00:05:56,555 245 00:05:56,555 --> 00:06:00,452 246 00:06:01,639 --> 00:06:04,519 247 00:06:04,519 --> 00:06:06,738 248 00:06:07,300 --> 00:06:10,769 249 00:06:10,769 --> 00:06:11,836 250 00:06:11,836 --> 00:06:15,000 251 00:06:15,200 --> 00:06:18,232 252 00:06:18,795 --> 00:06:20,090 253 00:06:20,090 --> 00:06:24,000 254 00:06:24,400 --> 00:06:28,183 255 00:06:28,183 --> 00:06:30,383 256 00:06:30,383 --> 00:06:32,100 257 00:06:32,100 --> 00:06:34,700 258 00:06:37,166 --> 00:06:39,416 259 00:06:39,697 --> 00:06:41,950 260 00:06:41,950 --> 00:06:44,930 261 00:06:44,930 --> 00:06:46,180 262 00:06:46,180 --> 00:06:47,956 263 00:06:47,956 --> 00:06:49,227 264 00:06:49,227 --> 00:06:50,700 265 00:06:51,600 --> 00:06:55,488 266 00:06:56,100 --> 00:06:58,121 267 00:06:58,121 --> 00:07:00,100 268 00:07:00,100 --> 00:07:01,900 269 00:07:02,100 --> 00:07:04,900 270 00:07:05,300 --> 00:07:08,921 271 00:07:08,921 --> 00:07:11,787 272 00:07:12,410 --> 00:07:14,975 273 00:07:15,925 --> 00:07:17,561 274 00:07:17,561 --> 00:07:21,700 275 00:07:21,930 --> 00:07:23,020 276 00:07:23,020 --> 00:07:24,600 277 00:07:24,600 --> 00:07:25,912 278 00:07:25,912 --> 00:07:28,100 279 00:07:28,100 --> 00:07:29,807 280 00:07:29,807 --> 00:07:30,908 281 00:07:30,908 --> 00:07:32,660 282 00:07:32,660 --> 00:07:34,182 283 00:07:34,700 --> 00:07:36,947 284 00:07:36,947 --> 00:07:39,350 285 00:07:41,354 --> 00:07:43,288 286 00:07:43,288 --> 00:07:46,112 287 00:07:46,112 --> 00:07:47,941 288 00:07:47,941 --> 00:07:51,415 289 00:07:51,415 --> 00:07:54,003 290 00:07:54,003 --> 00:07:55,636 291 00:07:55,636 --> 00:07:57,612 292 00:07:57,612 --> 00:07:59,579 293 00:08:00,900 --> 00:08:03,527 294 00:08:03,527 --> 00:08:06,916 295 00:08:06,916 --> 00:08:09,453 296 00:08:09,453 --> 00:08:12,200 297 00:08:12,400 --> 00:08:19,208 298 00:08:20,700 --> 00:08:23,734 299 00:08:23,734 --> 00:08:25,700 300 00:08:27,351 --> 00:08:31,600 301 00:08:31,800 --> 00:08:37,133 302 00:08:37,133 --> 00:08:40,948 303 00:08:41,500 --> 00:08:42,697 304 00:08:42,697 --> 00:08:45,300 305 00:08:45,300 --> 00:08:47,664 306 00:08:47,664 --> 00:08:49,610 307 00:08:49,610 --> 00:08:51,300 308 00:08:51,300 --> 00:08:53,700 309 00:08:54,100 --> 00:08:57,782 310 00:08:57,782 --> 00:09:00,269 311 00:09:00,269 --> 00:09:03,375 312 00:09:04,400 --> 00:09:06,531 313 00:09:06,531 --> 00:09:08,200 314 00:09:08,200 --> 00:09:10,000 315 00:09:10,900 --> 00:09:12,804 316 00:09:12,804 --> 00:09:14,501 317 00:09:14,501 --> 00:09:16,853 318 00:09:17,384 --> 00:09:19,933 319 00:09:19,933 --> 00:09:23,000 320 00:09:23,300 --> 00:09:25,839 321 00:09:25,839 --> 00:09:29,292 322 00:09:29,292 --> 00:09:32,200 323 00:09:32,944 --> 00:09:35,113 324 00:09:35,113 --> 00:09:37,600 325 00:09:38,708 --> 00:09:40,825 326 00:09:40,825 --> 00:09:42,600 327 00:09:42,900 --> 00:09:46,807 328 00:09:49,173 --> 00:09:51,100 329 00:09:51,944 --> 00:09:55,096 330 00:09:55,096 --> 00:09:57,100 331 00:09:57,400 --> 00:09:59,100 332 00:09:59,300 --> 00:10:01,096 333 00:10:02,900 --> 00:10:06,435 334 00:10:06,435 --> 00:10:09,424 335 00:10:10,206 --> 00:10:11,842 336 00:10:12,591 --> 00:10:14,033 337 00:10:14,403 --> 00:10:15,907 338 00:10:15,907 --> 00:10:17,289 339 00:10:17,289 --> 00:10:18,410 340 00:10:18,410 --> 00:10:20,001 341 00:10:20,001 --> 00:10:22,054 342 00:10:22,700 --> 00:10:26,823 343 00:10:27,555 --> 00:10:29,500 344 00:10:29,800 --> 00:10:31,028 345 00:10:31,591 --> 00:10:34,926 346 00:10:35,315 --> 00:10:37,151 347 00:10:38,761 --> 00:10:42,664 348 00:10:42,664 --> 00:10:46,560 349 00:10:48,800 --> 00:10:53,289 350 00:10:53,289 --> 00:10:55,700 351 00:10:56,300 --> 00:10:59,420 352 00:11:00,873 --> 00:11:02,055 353 00:11:02,900 --> 00:11:06,420 354 00:11:06,420 --> 00:11:09,040 355 00:11:09,040 --> 00:11:11,600 356 00:11:11,900 --> 00:11:13,453 357 00:11:13,808 --> 00:11:18,500 358 00:11:18,800 --> 00:11:20,885 359 00:11:20,885 --> 00:11:23,600 360 00:11:24,000 --> 00:11:26,800 361 00:11:26,800 --> 00:11:28,950 362 00:11:28,950 --> 00:11:31,210 363 00:11:31,210 --> 00:11:32,580 364 00:11:32,580 --> 00:11:34,684 365 00:11:34,684 --> 00:11:35,736 366 00:11:35,736 --> 00:11:37,000 367 00:11:37,300 --> 00:11:39,100 368 00:11:39,100 --> 00:11:41,051 369 00:11:41,051 --> 00:11:42,557 370 00:11:42,557 --> 00:11:44,449 371 00:11:44,449 --> 00:11:45,871 372 00:11:45,871 --> 00:11:47,141 373 00:11:47,141 --> 00:11:51,250 374 00:11:51,250 --> 00:11:52,394 375 00:11:52,394 --> 00:11:54,500 376 00:11:54,900 --> 00:11:57,608 377 00:11:57,608 --> 00:11:59,782 378 00:11:59,782 --> 00:12:04,234 379 00:12:04,234 --> 00:12:06,270 380 00:12:06,786 --> 00:12:09,864 381 00:12:09,864 --> 00:12:11,200 382 00:12:12,944 --> 00:12:14,934 383 00:12:14,934 --> 00:12:18,400 384 00:12:20,189 --> 00:12:23,656 385 00:12:24,700 --> 00:12:27,830 386 00:12:27,830 --> 00:12:29,622 387 00:12:29,622 --> 00:12:31,163 388 00:12:31,163 --> 00:12:34,000 389 00:12:34,900 --> 00:12:36,529 390 00:12:36,529 --> 00:12:40,013 391 00:12:40,582 --> 00:12:42,632 392 00:12:42,632 --> 00:12:43,887 393 00:12:43,887 --> 00:12:46,073 394 00:12:46,073 --> 00:12:49,100 395 00:12:49,800 --> 00:12:52,105 396 00:12:52,105 --> 00:12:53,573 397 00:12:53,573 --> 00:12:57,500 398 00:12:58,600 --> 00:13:00,159 399 00:13:00,159 --> 00:13:03,209 400 00:13:03,209 --> 00:13:06,090 401 00:13:06,549 --> 00:13:08,351 402 00:13:08,351 --> 00:13:10,979 403 00:13:11,198 --> 00:13:13,548 404 00:13:13,548 --> 00:13:16,850 405 00:13:16,850 --> 00:13:19,821 406 00:13:19,821 --> 00:13:23,870 407 00:13:24,424 --> 00:13:27,057 408 00:13:27,218 --> 00:13:30,132 409 00:13:30,132 --> 00:13:31,829 410 00:13:31,829 --> 00:13:33,486 411 00:13:33,486 --> 00:13:35,229 412 00:13:35,229 --> 00:13:37,189 413 00:13:37,189 --> 00:13:38,300 414 00:13:38,300 --> 00:13:41,267 415 00:13:41,267 --> 00:13:43,261 416 00:13:43,261 --> 00:13:46,465 417 00:13:48,657 --> 00:13:51,200 418 00:13:51,600 --> 00:13:54,187 419 00:13:54,187 --> 00:13:56,804 420 00:13:56,804 --> 00:13:58,207 421 00:13:58,207 --> 00:13:59,220 422 00:13:59,220 --> 00:14:00,679 423 00:14:00,679 --> 00:14:01,944 424 00:14:02,520 --> 00:14:03,963 425 00:14:03,963 --> 00:14:06,659 426 00:14:06,659 --> 00:14:08,000 427 00:14:08,000 --> 00:14:11,000 428 00:14:11,000 --> 00:14:12,395 429 00:14:12,395 --> 00:14:13,867 430 00:14:13,867 --> 00:14:15,914 431 00:14:15,914 --> 00:14:17,280 432 00:14:17,280 --> 00:14:18,305 433 00:14:18,305 --> 00:14:21,028 434 00:14:21,500 --> 00:14:24,425 435 00:14:24,698 --> 00:14:26,556 436 00:14:26,556 --> 00:14:27,968 437 00:14:27,968 --> 00:14:30,233 438 00:14:30,233 --> 00:14:32,489 439 00:14:32,489 --> 00:14:34,942 440 00:14:34,942 --> 00:14:36,440 441 00:14:36,440 --> 00:14:40,900 442 00:14:40,900 --> 00:14:42,200 443 00:14:42,200 --> 00:14:47,209 444 00:14:47,209 --> 00:14:49,900 445 00:14:51,000 --> 00:14:52,602 446 00:14:52,602 --> 00:14:55,410 447 00:14:55,410 --> 00:14:57,828 448 00:14:57,828 --> 00:14:59,463 449 00:14:59,463 --> 00:15:02,100 450 00:15:02,100 --> 00:15:03,800 451 00:15:03,800 --> 00:15:08,418 452 00:15:08,418 --> 00:15:09,519 453 00:15:09,519 --> 00:15:13,185 454 00:15:13,185 --> 00:15:14,502 455 00:15:15,700 --> 00:15:18,200 456 00:15:19,700 --> 00:15:21,984 457 00:15:21,984 --> 00:15:23,137 458 00:15:23,137 --> 00:15:25,135 459 00:15:25,805 --> 00:15:28,860 460 00:15:29,500 --> 00:15:31,900 461 00:15:32,200 --> 00:15:34,140 462 00:15:34,140 --> 00:15:37,507 463 00:15:38,100 --> 00:15:43,874 464 00:15:44,300 --> 00:15:46,700 465 00:15:47,300 --> 00:15:50,642 466 00:15:50,642 --> 00:15:53,900 467 00:15:53,900 --> 00:15:56,600 468 00:15:56,600 --> 00:15:58,600 469 00:15:59,000 --> 00:16:02,026 470 00:16:02,026 --> 00:16:06,648 471 00:16:06,648 --> 00:16:08,328 472 00:16:08,328 --> 00:16:11,300 473 00:16:11,300 --> 00:16:12,819 474 00:16:12,819 --> 00:16:14,753 475 00:16:14,753 --> 00:16:18,191 476 00:16:18,191 --> 00:16:20,400 477 00:16:20,950 --> 00:16:22,394 478 00:16:22,500 --> 00:16:24,999 479 00:16:24,999 --> 00:16:26,212 480 00:16:26,425 --> 00:16:27,993 481 00:16:27,993 --> 00:16:33,200 482 00:16:34,100 --> 00:16:35,710 483 00:16:35,920 --> 00:16:37,148 484 00:16:37,148 --> 00:16:40,066 485 00:16:40,066 --> 00:16:43,316 486 00:16:43,316 --> 00:16:45,100 487 00:16:45,100 --> 00:16:46,392 488 00:16:46,392 --> 00:16:49,700 489 00:16:50,000 --> 00:16:52,020 490 00:16:52,020 --> 00:16:53,526 491 00:16:53,526 --> 00:16:55,184 492 00:16:55,184 --> 00:16:57,400 493 00:16:57,976 --> 00:17:00,350 494 00:17:00,350 --> 00:17:03,102 495 00:17:03,102 --> 00:17:05,671 496 00:17:05,671 --> 00:17:08,610 497 00:17:08,610 --> 00:17:09,811 498 00:17:09,811 --> 00:17:12,000 499 00:17:12,400 --> 00:17:15,309 500 00:17:15,309 --> 00:17:17,292 501 00:17:17,292 --> 00:17:19,100 502 00:17:19,500 --> 00:17:21,849 503 00:17:21,849 --> 00:17:23,155 504 00:17:23,155 --> 00:17:25,063 505 00:17:25,063 --> 00:17:27,100 506 00:17:27,872 --> 00:17:29,965 507 00:17:29,965 --> 00:17:32,400 508 00:17:32,400 --> 00:17:35,100 509 00:17:35,100 --> 00:17:36,625 510 00:17:36,625 --> 00:17:40,573 511 00:17:40,573 --> 00:17:43,620 512 00:17:43,620 --> 00:17:45,797 513 00:17:45,797 --> 00:17:47,733 514 00:17:47,733 --> 00:17:50,100 515 00:17:50,100 --> 00:17:52,983 516 00:17:52,983 --> 00:17:55,700 517 00:17:55,700 --> 00:17:58,000 518 00:17:58,000 --> 00:18:01,229 519 00:18:01,229 --> 00:18:03,300 520 00:18:03,600 --> 00:18:05,985 521 00:18:05,985 --> 00:18:07,668 522 00:18:07,668 --> 00:18:10,694 523 00:18:10,694 --> 00:18:12,800 524 00:18:13,000 --> 00:18:16,780 525 00:18:16,780 --> 00:18:18,517 526 00:18:18,517 --> 00:18:20,938 527 00:18:20,938 --> 00:18:22,139 528 00:18:22,139 --> 00:18:27,309 529 00:18:27,309 --> 00:18:28,676 530 00:18:28,900 --> 00:18:30,492 531 00:18:30,492 --> 00:18:33,782 532 00:18:33,782 --> 00:18:35,228 533 00:18:35,529 --> 00:18:39,500 534 00:18:39,500 --> 00:18:41,824 535 00:18:41,824 --> 00:18:44,000 536 00:18:44,000 --> 00:18:47,065 537 00:18:47,065 --> 00:18:48,800 538 00:18:49,000 --> 00:18:52,964 539 00:18:53,504 --> 00:18:56,000 540 00:18:56,300 --> 00:18:58,791 541 00:18:58,791 --> 00:19:01,200 542 00:19:01,500 --> 00:19:02,649 543 00:19:02,649 --> 00:19:06,300 544 00:19:06,500 --> 00:19:08,181 545 00:19:08,181 --> 00:19:09,779 546 00:19:09,779 --> 00:19:11,489 547 00:19:11,489 --> 00:19:12,741 548 00:19:12,741 --> 00:19:14,274 549 00:19:14,274 --> 00:19:17,067 550 00:19:17,067 --> 00:19:20,148 551 00:19:20,349 --> 00:19:21,388 552 00:19:21,388 --> 00:19:23,673 553 00:19:23,673 --> 00:19:25,801 554 00:19:25,801 --> 00:19:28,603 555 00:19:29,900 --> 00:19:32,704 556 00:19:32,704 --> 00:19:35,816 557 00:19:35,816 --> 00:19:37,500 558 00:19:37,500 --> 00:19:40,223 559 00:19:40,223 --> 00:19:41,388 560 00:19:41,388 --> 00:19:43,090 561 00:19:43,090 --> 00:19:45,394 562 00:19:45,394 --> 00:19:46,630 563 00:19:47,100 --> 00:19:49,300 564 00:19:49,300 --> 00:19:52,932 565 00:19:52,932 --> 00:19:54,414 566 00:19:54,414 --> 00:19:56,400 567 00:19:57,000 --> 00:20:00,297 568 00:20:00,297 --> 00:20:03,539 569 00:20:03,539 --> 00:20:06,100 570 00:20:06,600 --> 00:20:08,720 571 00:20:08,720 --> 00:20:10,174 572 00:20:10,174 --> 00:20:11,633 573 00:20:12,926 --> 00:20:14,419 574 00:20:14,419 --> 00:20:16,833 575 00:20:16,833 --> 00:20:18,012 576 00:20:18,012 --> 00:20:20,500 577 00:20:20,500 --> 00:20:23,100 578 00:20:23,600 --> 00:20:26,569 579 00:20:26,569 --> 00:20:28,550 580 00:20:28,550 --> 00:20:30,535