9:59:59.000,9:59:59.000
1[br]00:00:04,190 --> 00:00:09,250[br]So this video, we're want to talk about asking questions.

9:59:59.000,9:59:59.000
2[br]00:00:09,250 --> 00:00:16,020[br]What makes a good question? How does a question relate to the broader context of what we're trying to do in this class?

9:59:59.000,9:59:59.000
3[br]00:00:16,020 --> 00:00:21,130[br]The learning outcomes for this video are few to understand what makes a good question.

9:59:59.000,9:59:59.000
4[br]00:00:21,130 --> 00:00:26,470[br]Understand how it relates to goals and analysis and start to think about data for a question.

9:59:59.000,9:59:59.000
5[br]00:00:26,470 --> 00:00:35,790[br]We're also going to introduce a key term operationalization that is going to come up throughout the rest of the class.

9:59:59.000,9:59:59.000
6[br]00:00:35,790 --> 00:00:44,400[br]To set the stage, I want to review our definition of data science that I introduced in the class introduction video that we're learning

9:59:59.000,9:59:59.000
7[br]00:00:44,400 --> 00:00:52,500[br]about how to use data to provide quantitative insights on questions of scientific business or social interest.

9:59:59.000,9:59:59.000
8[br]00:00:52,500 --> 00:01:00,120[br]But in order to do that effectively, we need to be able to write good questions, refine those questions,

9:59:59.000,9:59:59.000
9[br]00:01:00,120 --> 00:01:11,370[br]connect them both to the data we might be able to use to shed these quantitative insights and to the goals,

9:59:59.000,9:59:59.000
10[br]00:01:11,370 --> 00:01:17,870[br]the business purposes or scientific purposes for which we're asking the questions in the first place.

9:59:59.000,9:59:59.000
11[br]00:01:17,870 --> 00:01:23,060[br]So I want to work through this with you with an example.

9:59:59.000,9:59:59.000
12[br]00:01:23,060 --> 00:01:28,010[br]So suppose in the Boys State Computer Science Department, we have our introductory classes.

9:59:59.000,9:59:59.000
13[br]00:01:28,010 --> 00:01:32,750[br]Yes. One twenty one to twenty one. Three twenty one. Suppose we make some change to see.

9:59:59.000,9:59:59.000
14[br]00:01:32,750 --> 00:01:40,030[br]Yes. Twenty one. Like we change the way we do the assignments. And we want to assess whether this new change improved.

9:59:59.000,9:59:59.000
15[br]00:01:40,030 --> 00:01:46,070[br]C. S 121. So we have a business purpose here of we're making a change to one of our courses.

9:59:59.000,9:59:59.000
16[br]00:01:46,070 --> 00:01:50,540[br]And we want to see if that change is improving the course in some way.

9:59:59.000,9:59:59.000
17[br]00:01:50,540 --> 00:01:56,580[br]But in order to do that, we need to identify a number of things, such as what does it mean to improve C.

9:59:59.000,9:59:59.000
18[br]00:01:56,580 --> 00:02:04,510[br]S one twenty one? What data could we use to try to inform this assessment of whether we improved?

9:59:59.000,9:59:59.000
19[br]00:02:04,510 --> 00:02:11,390[br]Yes, 121. And what could we do with that data to measure improvement?

9:59:59.000,9:59:59.000
20[br]00:02:11,390 --> 00:02:17,540[br]And. So this process is called operationalization.

9:59:59.000,9:59:59.000
21[br]00:02:17,540 --> 00:02:27,620[br]We have a goal. Assess whether we improved 121. That, in turn is in service of the broader goal of delivering a high quality undergraduate education.

9:59:59.000,9:59:59.000
22[br]00:02:27,620 --> 00:02:34,760[br]Then we've refined through intermediate questions. I'm going to show some of those in a bit to determine a specific measurement to take.

9:59:59.000,9:59:59.000
23[br]00:02:34,760 --> 00:02:39,470[br]And at the end of the day, if we are fully operationalize the goal or a question,

9:59:59.000,9:59:59.000
24[br]00:02:39,470 --> 00:02:45,860[br]we know precisely what data we're going to collect or has been collected and how what measurement

9:59:59.000,9:59:59.000
25[br]00:02:45,860 --> 00:02:51,140[br]or measurements we're going to compute over that data in order to try to answer our question,

9:59:59.000,9:59:59.000
26[br]00:02:51,140 --> 00:02:55,940[br]we use the term in a couple of senses. First, operationalize can be a verb.

9:59:59.000,9:59:59.000
27[br]00:02:55,940 --> 00:03:03,290[br]It's the process of doing this operationalization. Then, as a tense of the verb operationalization is also a noun.

9:59:59.000,9:59:59.000
28[br]00:03:03,290 --> 00:03:05,330[br]And it's the result of this process.

9:59:59.000,9:59:59.000
29[br]00:03:05,330 --> 00:03:15,500[br]So the specific measurement and analysis that we're going to do over specific data can be called an operationalization of the question.

9:59:59.000,9:59:59.000
30[br]00:03:15,500 --> 00:03:19,760[br]So we have our goal of assessing whether some change improves.

9:59:59.000,9:59:59.000
31[br]00:03:19,760 --> 00:03:24,290[br]Yes, 121. We can ask an intermediate question.

9:59:59.000,9:59:59.000
32[br]00:03:24,290 --> 00:03:29,900[br]OK, so what does it mean to improve it? Well, students are better prepared to go excel in the workplace.

9:59:59.000,9:59:59.000
33[br]00:03:29,900 --> 00:03:36,650[br]Well, it's a while until this is the freshman class. It's a while until the students are going out and the job market.

9:59:59.000,9:59:59.000
34[br]00:03:36,650 --> 00:03:42,950[br]Or we have for information on how well equipped they were. So can we ask a shorter term question?

9:59:59.000,9:59:59.000
35[br]00:03:42,950 --> 00:03:50,030[br]That's going to help us get to that. Are students better prepared for the next class?

9:59:59.000,9:59:59.000
36[br]00:03:50,030 --> 00:03:57,100[br]And we call this intermediate question a proxy.

9:59:59.000,9:59:59.000
37[br]00:03:57,100 --> 00:04:05,560[br]So if our goal is better, prepare them for doing their work, they're doing the work we're training them for the proxy can be.

9:59:59.000,9:59:59.000
38[br]00:04:05,560 --> 00:04:09,280[br]Well, are they better prepared for the next class?

9:59:59.000,9:59:59.000
39[br]00:04:09,280 --> 00:04:21,010[br]So questions don't have one level and there's a there's a path here between goal our goal, improve education, deliver a high quality education,

9:59:59.000,9:59:59.000
40[br]00:04:21,010 --> 00:04:30,160[br]the subgoal of assess whether this change that was intended to improve the educational effectiveness of our introductory

9:59:59.000,9:59:59.000
41[br]00:04:30,160 --> 00:04:38,620[br]programing class actually did so to get all the way down to the data that we can use in order to try to measure it.

9:59:59.000,9:59:59.000
42[br]00:04:38,620 --> 00:04:43,300[br]We can also have multiple levels of questions, as we've already seen well.

9:59:59.000,9:59:59.000
43[br]00:04:43,300 --> 00:04:47,170[br]Are they prepared for for their work? Well, we can't. That's a long timeframe.

9:59:59.000,9:59:59.000
44[br]00:04:47,170 --> 00:04:53,650[br]It's difficult to measure that on the timeframe we need in order to iterate on the on class structures.

9:59:59.000,9:59:59.000
45[br]00:04:53,650 --> 00:04:58,180[br]So we use this that we step down one level. We use this proxy.

9:59:59.000,9:59:59.000
46[br]00:04:58,180 --> 00:05:01,630[br]Are they better prepared for the next class?

9:59:59.000,9:59:59.000
47[br]00:05:01,630 --> 00:05:11,320[br]So if we want to think about the quality of our questions, like we need a way to assess whether a question is good.

9:59:59.000,9:59:59.000
48[br]00:05:11,320 --> 00:05:14,210[br]And there's a couple of ways we do that. One is looking upward.

9:59:59.000,9:59:59.000
49[br]00:05:14,210 --> 00:05:21,910[br]So the question should advance the goal and we should be able to look at the goal and look at the question and say yes.

9:59:59.000,9:59:59.000
50[br]00:05:21,910 --> 00:05:26,950[br]Answering this question does move us forward in this goal.

9:59:59.000,9:59:59.000
51[br]00:05:26,950 --> 00:05:30,760[br]No one question is going to be the complete answer to our goal.

9:59:59.000,9:59:59.000
52[br]00:05:30,760 --> 00:05:35,710[br]But our students, better prepared for the next class, moves us one step closer.

9:59:59.000,9:59:59.000
53[br]00:05:35,710 --> 00:05:40,300[br]We can say yes, if we if students are better prepared for the next class,

9:59:59.000,9:59:59.000
54[br]00:05:40,300 --> 00:05:46,360[br]that is probably evidence that we have improved the effectiveness of the introductory class.

9:59:59.000,9:59:59.000
55[br]00:05:46,360 --> 00:05:52,660[br]Also, though, carrying out the analysis should answer the question.

9:59:59.000,9:59:59.000
56[br]00:05:52,660 --> 00:05:58,540[br]We want to work our questions down to the point where we have a question that's specific.

9:59:59.000,9:59:59.000
57[br]00:05:58,540 --> 00:06:07,240[br]We we can it's clear that the question will advance either the top level goal or a higher level question that in turn advances the goal.

9:59:59.000,9:59:59.000
58[br]00:06:07,240 --> 00:06:11,110[br]But also it's specific enough that we can look at a data analysis plan.

9:59:59.000,9:59:59.000
59[br]00:06:11,110 --> 00:06:17,140[br]Here's the data we're going to use. Here's the measurements we're going to take. Here's the analysis we're going to perform and we can say, yes,

9:59:59.000,9:59:59.000
60[br]00:06:17,140 --> 00:06:24,220[br]doing this data analysis plan will answer this question or at least answer the question in a useful sense.

9:59:59.000,9:59:59.000
61[br]00:06:24,220 --> 00:06:31,240[br]And so if we can make those connections that we can see, doing the analysis will answer the question, answer the question.

9:59:59.000,9:59:59.000
62[br]00:06:31,240 --> 00:06:35,350[br]Answering the question will advance the goal. Then we have a connection.

9:59:59.000,9:59:59.000
63[br]00:06:35,350 --> 00:06:41,620[br]We have a connectedness between the analysis and the data that we can actually do.

9:59:59.000,9:59:59.000
64[br]00:06:41,620 --> 00:06:48,720[br]And. The question or the goal that we're trying to advance through this data analysis.

9:59:59.000,9:59:59.000
65[br]00:06:48,720 --> 00:06:57,030[br]So a fully operationalized question is going to be specific and it's going to be answerable and with the available data.

9:59:59.000,9:59:59.000
66[br]00:06:57,030 --> 00:07:01,080[br]Now, there are lots of useful questions that we can't answer with available data.

9:59:59.000,9:59:59.000
67[br]00:07:01,080 --> 00:07:11,080[br]That does not mean they're bad or we should ignore them. They're incredibly useful for contextualizing the limits of a data analysis that we do.

9:59:59.000,9:59:59.000
68[br]00:07:11,080 --> 00:07:14,290[br]We have a data analysis. It can answer one question that will advance the goal.

9:59:59.000,9:59:59.000
69[br]00:07:14,290 --> 00:07:18,760[br]There are three other questions related to the goal that cannot be answered by our analysis.

9:59:59.000,9:59:59.000
70[br]00:07:18,760 --> 00:07:23,080[br]Well, that's useful in our report to talk about the limitations. Well, we can answer this question.

9:59:59.000,9:59:59.000
71[br]00:07:23,080 --> 00:07:27,880[br]We can't answer these others. Maybe we can think about how to how to answer those others questions.

9:59:59.000,9:59:59.000
72[br]00:07:27,880 --> 00:07:31,130[br]But when we're trying to get down to a question that we can answer with data.

9:59:59.000,9:59:59.000
73[br]00:07:31,130 --> 00:07:39,370[br]And remember, we're talking about data sciences, quantitative insights into these questions.

9:59:59.000,9:59:59.000
74[br]00:07:39,370 --> 00:07:45,580[br]We want to see, can we actually answer the question with data? And can we match the analysis plan to the question to the goal.

9:59:59.000,9:59:59.000
75[br]00:07:45,580 --> 00:07:53,830[br]So to go back to our example of trying to measure the effectiveness of improving one twenty one, are students better prepared for the next class?

9:59:59.000,9:59:59.000
76[br]00:07:53,830 --> 00:07:57,870[br]Well, we can make that more specific. Are they more likely to pass?

9:59:59.000,9:59:59.000
77[br]00:07:57,870 --> 00:08:01,960[br]Yes. To twenty one. Now we have a very specific question.

9:59:59.000,9:59:59.000
78[br]00:08:01,960 --> 00:08:07,930[br]We can answer it with the student grades from six to twenty one. We can look at students who took our class.

9:59:59.000,9:59:59.000
79[br]00:08:07,930 --> 00:08:11,950[br]Our new C. S one twenty one and took our old C as one.

9:59:59.000,9:59:59.000
80[br]00:08:11,950 --> 00:08:15,940[br]And we can compare the pass rates. Now there are many caveats.

9:59:59.000,9:59:59.000
81[br]00:08:15,940 --> 00:08:21,640[br]There are a lot of challenges to doing this properly. It can only measure one piece of what's going on.

9:59:59.000,9:59:59.000
82[br]00:08:21,640 --> 00:08:25,300[br]But it's a specific question that we can answer with data.

9:59:59.000,9:59:59.000
83[br]00:08:25,300 --> 00:08:33,280[br]Our students in the new version of our intro class more or less likely to pass the next class,

9:59:59.000,9:59:59.000
84[br]00:08:33,280 --> 00:08:37,090[br]will get to talk more about this question in the next video.

9:59:59.000,9:59:59.000
85[br]00:08:37,090 --> 00:08:43,270[br]Now, to get to this kind of a question, I've given you the example and work through it here.

9:59:59.000,9:59:59.000
86[br]00:08:43,270 --> 00:08:51,310[br]In practice, you're going to need to work with your boss, your client, your advisor, other stakeholders,

9:59:59.000,9:59:59.000
87[br]00:08:51,310 --> 00:08:58,560[br]whoever is going to be acting on the results of your data analysis, which may be yourself.

9:59:59.000,9:59:59.000
88[br]00:08:58,560 --> 00:09:05,190[br]To get to these operation, to get to these fully operationalized questions, they're going to have goals.

9:59:59.000,9:59:59.000
89[br]00:09:05,190 --> 00:09:11,130[br]They may have some some high level questions, they may have some specific questions that can't map to the data.

9:59:59.000,9:59:59.000
90[br]00:09:11,130 --> 00:09:17,370[br]One of the key ways to be able to do this refinement is through clarifying questions such as.

9:59:59.000,9:59:59.000
91[br]00:09:17,370 --> 00:09:21,240[br]So if if the department chair came to you and said,

9:59:59.000,9:59:59.000
92[br]00:09:21,240 --> 00:09:28,350[br]I would like you to help me measure the effect of this improvement to see us one twenty one, well, then we can ask questions.

9:59:59.000,9:59:59.000
93[br]00:09:28,350 --> 00:09:34,520[br]What do we mean by improve? What would be evidence that we did improve?

9:59:59.000,9:59:59.000
94[br]00:09:34,520 --> 00:09:39,500[br]Six one twenty one. And so we're gonna have practice in the synchronous time.

9:59:59.000,9:59:59.000
95[br]00:09:39,500 --> 00:09:44,360[br]That's one of the things we're gonna do this week in thinking about clarifying questions.

9:59:59.000,9:59:59.000
96[br]00:09:44,360 --> 00:09:48,050[br]But these clarifying questions that you can ask to your client.

9:59:59.000,9:59:59.000
97[br]00:09:48,050 --> 00:09:52,310[br]We're going to use the term client generally for whoever is you're doing the data

9:59:59.000,9:59:59.000
98[br]00:09:52,310 --> 00:09:57,670[br]analysis for to figure out what they actually want and what you can do with the data.

9:59:59.000,9:59:59.000
99[br]00:09:57,670 --> 00:10:05,360[br]That's going to advance their goals. So to wrap up, there are multiple layers to translate between our high level goals,

9:59:59.000,9:59:59.000
100[br]00:10:05,360 --> 00:10:11,030[br]deliver a high quality undergraduate education and what we can actually do with data

9:59:59.000,9:59:59.000
101[br]00:10:11,030 --> 00:10:16,850[br]measure whether this change increased students ability to pass the next class.

9:59:59.000,9:59:59.000
102[br]00:10:16,850 --> 00:10:26,570[br]Questions bridge this gap and we can have multiple layers of questions in order to get from high level goal to something we can do with data.

9:59:59.000,9:59:59.000
103[br]00:10:26,570 --> 00:10:35,933[br]You're gonna be doing this a lot through the rest of the semester.

9:59:59.000,9:59:59.000