[Script Info]
Title: 
[Events]
Format: Layer, Start, End, Style, Name, MarginL, MarginR, MarginV, Effect, Text
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,1\N00:00:04,190 --> 00:00:09,250\NSo this video, we're want to talk about asking questions.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,2\N00:00:09,250 --> 00:00:16,020\NWhat makes a good question? How does a question relate to the broader context of what we're trying to do in this class?
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,3\N00:00:16,020 --> 00:00:21,130\NThe learning outcomes for this video are few to understand what makes a good question.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,4\N00:00:21,130 --> 00:00:26,470\NUnderstand how it relates to goals and analysis and start to think about data for a question.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,5\N00:00:26,470 --> 00:00:35,790\NWe're also going to introduce a key term operationalization that is going to come up throughout the rest of the class.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,6\N00:00:35,790 --> 00:00:44,400\NTo set the stage, I want to review our definition of data science that I introduced in the class introduction video that we're learning
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,7\N00:00:44,400 --> 00:00:52,500\Nabout how to use data to provide quantitative insights on questions of scientific business or social interest.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,8\N00:00:52,500 --> 00:01:00,120\NBut in order to do that effectively, we need to be able to write good questions, refine those questions,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,9\N00:01:00,120 --> 00:01:11,370\Nconnect them both to the data we might be able to use to shed these quantitative insights and to the goals,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,10\N00:01:11,370 --> 00:01:17,870\Nthe business purposes or scientific purposes for which we're asking the questions in the first place.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,11\N00:01:17,870 --> 00:01:23,060\NSo I want to work through this with you with an example.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,12\N00:01:23,060 --> 00:01:28,010\NSo suppose in the Boys State Computer Science Department, we have our introductory classes.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,13\N00:01:28,010 --> 00:01:32,750\NYes. One twenty one to twenty one. Three twenty one. Suppose we make some change to see.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,14\N00:01:32,750 --> 00:01:40,030\NYes. Twenty one. Like we change the way we do the assignments. And we want to assess whether this new change improved.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,15\N00:01:40,030 --> 00:01:46,070\NC. S 121. So we have a business purpose here of we're making a change to one of our courses.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,16\N00:01:46,070 --> 00:01:50,540\NAnd we want to see if that change is improving the course in some way.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,17\N00:01:50,540 --> 00:01:56,580\NBut in order to do that, we need to identify a number of things, such as what does it mean to improve C.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,18\N00:01:56,580 --> 00:02:04,510\NS one twenty one? What data could we use to try to inform this assessment of whether we improved?
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,19\N00:02:04,510 --> 00:02:11,390\NYes, 121. And what could we do with that data to measure improvement?
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,20\N00:02:11,390 --> 00:02:17,540\NAnd. So this process is called operationalization.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,21\N00:02:17,540 --> 00:02:27,620\NWe have a goal. Assess whether we improved 121. That, in turn is in service of the broader goal of delivering a high quality undergraduate education.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,22\N00:02:27,620 --> 00:02:34,760\NThen we've refined through intermediate questions. I'm going to show some of those in a bit to determine a specific measurement to take.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,23\N00:02:34,760 --> 00:02:39,470\NAnd at the end of the day, if we are fully operationalize the goal or a question,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,24\N00:02:39,470 --> 00:02:45,860\Nwe know precisely what data we're going to collect or has been collected and how what measurement
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,25\N00:02:45,860 --> 00:02:51,140\Nor measurements we're going to compute over that data in order to try to answer our question,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,26\N00:02:51,140 --> 00:02:55,940\Nwe use the term in a couple of senses. First, operationalize can be a verb.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,27\N00:02:55,940 --> 00:03:03,290\NIt's the process of doing this operationalization. Then, as a tense of the verb operationalization is also a noun.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,28\N00:03:03,290 --> 00:03:05,330\NAnd it's the result of this process.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,29\N00:03:05,330 --> 00:03:15,500\NSo the specific measurement and analysis that we're going to do over specific data can be called an operationalization of the question.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,30\N00:03:15,500 --> 00:03:19,760\NSo we have our goal of assessing whether some change improves.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,31\N00:03:19,760 --> 00:03:24,290\NYes, 121. We can ask an intermediate question.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,32\N00:03:24,290 --> 00:03:29,900\NOK, so what does it mean to improve it? Well, students are better prepared to go excel in the workplace.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,33\N00:03:29,900 --> 00:03:36,650\NWell, it's a while until this is the freshman class. It's a while until the students are going out and the job market.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,34\N00:03:36,650 --> 00:03:42,950\NOr we have for information on how well equipped they were. So can we ask a shorter term question?
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,35\N00:03:42,950 --> 00:03:50,030\NThat's going to help us get to that. Are students better prepared for the next class?
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,36\N00:03:50,030 --> 00:03:57,100\NAnd we call this intermediate question a proxy.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,37\N00:03:57,100 --> 00:04:05,560\NSo if our goal is better, prepare them for doing their work, they're doing the work we're training them for the proxy can be.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,38\N00:04:05,560 --> 00:04:09,280\NWell, are they better prepared for the next class?
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,39\N00:04:09,280 --> 00:04:21,010\NSo questions don't have one level and there's a there's a path here between goal our goal, improve education, deliver a high quality education,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,40\N00:04:21,010 --> 00:04:30,160\Nthe subgoal of assess whether this change that was intended to improve the educational effectiveness of our introductory
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,41\N00:04:30,160 --> 00:04:38,620\Nprograming class actually did so to get all the way down to the data that we can use in order to try to measure it.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,42\N00:04:38,620 --> 00:04:43,300\NWe can also have multiple levels of questions, as we've already seen well.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,43\N00:04:43,300 --> 00:04:47,170\NAre they prepared for for their work? Well, we can't. That's a long timeframe.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,44\N00:04:47,170 --> 00:04:53,650\NIt's difficult to measure that on the timeframe we need in order to iterate on the on class structures.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,45\N00:04:53,650 --> 00:04:58,180\NSo we use this that we step down one level. We use this proxy.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,46\N00:04:58,180 --> 00:05:01,630\NAre they better prepared for the next class?
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,47\N00:05:01,630 --> 00:05:11,320\NSo if we want to think about the quality of our questions, like we need a way to assess whether a question is good.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,48\N00:05:11,320 --> 00:05:14,210\NAnd there's a couple of ways we do that. One is looking upward.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,49\N00:05:14,210 --> 00:05:21,910\NSo the question should advance the goal and we should be able to look at the goal and look at the question and say yes.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,50\N00:05:21,910 --> 00:05:26,950\NAnswering this question does move us forward in this goal.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,51\N00:05:26,950 --> 00:05:30,760\NNo one question is going to be the complete answer to our goal.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,52\N00:05:30,760 --> 00:05:35,710\NBut our students, better prepared for the next class, moves us one step closer.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,53\N00:05:35,710 --> 00:05:40,300\NWe can say yes, if we if students are better prepared for the next class,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,54\N00:05:40,300 --> 00:05:46,360\Nthat is probably evidence that we have improved the effectiveness of the introductory class.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,55\N00:05:46,360 --> 00:05:52,660\NAlso, though, carrying out the analysis should answer the question.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,56\N00:05:52,660 --> 00:05:58,540\NWe want to work our questions down to the point where we have a question that's specific.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,57\N00:05:58,540 --> 00:06:07,240\NWe we can it's clear that the question will advance either the top level goal or a higher level question that in turn advances the goal.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,58\N00:06:07,240 --> 00:06:11,110\NBut also it's specific enough that we can look at a data analysis plan.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,59\N00:06:11,110 --> 00:06:17,140\NHere's the data we're going to use. Here's the measurements we're going to take. Here's the analysis we're going to perform and we can say, yes,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,60\N00:06:17,140 --> 00:06:24,220\Ndoing this data analysis plan will answer this question or at least answer the question in a useful sense.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,61\N00:06:24,220 --> 00:06:31,240\NAnd so if we can make those connections that we can see, doing the analysis will answer the question, answer the question.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,62\N00:06:31,240 --> 00:06:35,350\NAnswering the question will advance the goal. Then we have a connection.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,63\N00:06:35,350 --> 00:06:41,620\NWe have a connectedness between the analysis and the data that we can actually do.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,64\N00:06:41,620 --> 00:06:48,720\NAnd. The question or the goal that we're trying to advance through this data analysis.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,65\N00:06:48,720 --> 00:06:57,030\NSo a fully operationalized question is going to be specific and it's going to be answerable and with the available data.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,66\N00:06:57,030 --> 00:07:01,080\NNow, there are lots of useful questions that we can't answer with available data.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,67\N00:07:01,080 --> 00:07:11,080\NThat does not mean they're bad or we should ignore them. They're incredibly useful for contextualizing the limits of a data analysis that we do.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,68\N00:07:11,080 --> 00:07:14,290\NWe have a data analysis. It can answer one question that will advance the goal.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,69\N00:07:14,290 --> 00:07:18,760\NThere are three other questions related to the goal that cannot be answered by our analysis.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,70\N00:07:18,760 --> 00:07:23,080\NWell, that's useful in our report to talk about the limitations. Well, we can answer this question.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,71\N00:07:23,080 --> 00:07:27,880\NWe can't answer these others. Maybe we can think about how to how to answer those others questions.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,72\N00:07:27,880 --> 00:07:31,130\NBut when we're trying to get down to a question that we can answer with data.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,73\N00:07:31,130 --> 00:07:39,370\NAnd remember, we're talking about data sciences, quantitative insights into these questions.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,74\N00:07:39,370 --> 00:07:45,580\NWe want to see, can we actually answer the question with data? And can we match the analysis plan to the question to the goal.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,75\N00:07:45,580 --> 00:07:53,830\NSo to go back to our example of trying to measure the effectiveness of improving one twenty one, are students better prepared for the next class?
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,76\N00:07:53,830 --> 00:07:57,870\NWell, we can make that more specific. Are they more likely to pass?
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,77\N00:07:57,870 --> 00:08:01,960\NYes. To twenty one. Now we have a very specific question.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,78\N00:08:01,960 --> 00:08:07,930\NWe can answer it with the student grades from six to twenty one. We can look at students who took our class.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,79\N00:08:07,930 --> 00:08:11,950\NOur new C. S one twenty one and took our old C as one.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,80\N00:08:11,950 --> 00:08:15,940\NAnd we can compare the pass rates. Now there are many caveats.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,81\N00:08:15,940 --> 00:08:21,640\NThere are a lot of challenges to doing this properly. It can only measure one piece of what's going on.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,82\N00:08:21,640 --> 00:08:25,300\NBut it's a specific question that we can answer with data.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,83\N00:08:25,300 --> 00:08:33,280\NOur students in the new version of our intro class more or less likely to pass the next class,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,84\N00:08:33,280 --> 00:08:37,090\Nwill get to talk more about this question in the next video.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,85\N00:08:37,090 --> 00:08:43,270\NNow, to get to this kind of a question, I've given you the example and work through it here.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,86\N00:08:43,270 --> 00:08:51,310\NIn practice, you're going to need to work with your boss, your client, your advisor, other stakeholders,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,87\N00:08:51,310 --> 00:08:58,560\Nwhoever is going to be acting on the results of your data analysis, which may be yourself.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,88\N00:08:58,560 --> 00:09:05,190\NTo get to these operation, to get to these fully operationalized questions, they're going to have goals.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,89\N00:09:05,190 --> 00:09:11,130\NThey may have some some high level questions, they may have some specific questions that can't map to the data.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,90\N00:09:11,130 --> 00:09:17,370\NOne of the key ways to be able to do this refinement is through clarifying questions such as.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,91\N00:09:17,370 --> 00:09:21,240\NSo if if the department chair came to you and said,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,92\N00:09:21,240 --> 00:09:28,350\NI would like you to help me measure the effect of this improvement to see us one twenty one, well, then we can ask questions.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,93\N00:09:28,350 --> 00:09:34,520\NWhat do we mean by improve? What would be evidence that we did improve?
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,94\N00:09:34,520 --> 00:09:39,500\NSix one twenty one. And so we're gonna have practice in the synchronous time.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,95\N00:09:39,500 --> 00:09:44,360\NThat's one of the things we're gonna do this week in thinking about clarifying questions.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,96\N00:09:44,360 --> 00:09:48,050\NBut these clarifying questions that you can ask to your client.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,97\N00:09:48,050 --> 00:09:52,310\NWe're going to use the term client generally for whoever is you're doing the data
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,98\N00:09:52,310 --> 00:09:57,670\Nanalysis for to figure out what they actually want and what you can do with the data.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,99\N00:09:57,670 --> 00:10:05,360\NThat's going to advance their goals. So to wrap up, there are multiple layers to translate between our high level goals,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,100\N00:10:05,360 --> 00:10:11,030\Ndeliver a high quality undergraduate education and what we can actually do with data
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,101\N00:10:11,030 --> 00:10:16,850\Nmeasure whether this change increased students ability to pass the next class.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,102\N00:10:16,850 --> 00:10:26,570\NQuestions bridge this gap and we can have multiple layers of questions in order to get from high level goal to something we can do with data.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,103\N00:10:26,570 --> 00:10:35,933\NYou're gonna be doing this a lot through the rest of the semester.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,