WEBVTT

99:59:59.999 --> 99:59:59.999
1
00:00:04,190 --> 00:00:09,250
So this video, we're want to talk about asking questions.

99:59:59.999 --> 99:59:59.999
2
00:00:09,250 --> 00:00:16,020
What makes a good question? How does a question relate to the broader context of what we're trying to do in this class?

99:59:59.999 --> 99:59:59.999
3
00:00:16,020 --> 00:00:21,130
The learning outcomes for this video are few to understand what makes a good question.

99:59:59.999 --> 99:59:59.999
4
00:00:21,130 --> 00:00:26,470
Understand how it relates to goals and analysis and start to think about data for a question.

99:59:59.999 --> 99:59:59.999
5
00:00:26,470 --> 00:00:35,790
We're also going to introduce a key term operationalization that is going to come up throughout the rest of the class.

99:59:59.999 --> 99:59:59.999
6
00:00:35,790 --> 00:00:44,400
To set the stage, I want to review our definition of data science that I introduced in the class introduction video that we're learning

99:59:59.999 --> 99:59:59.999
7
00:00:44,400 --> 00:00:52,500
about how to use data to provide quantitative insights on questions of scientific business or social interest.

99:59:59.999 --> 99:59:59.999
8
00:00:52,500 --> 00:01:00,120
But in order to do that effectively, we need to be able to write good questions, refine those questions,

99:59:59.999 --> 99:59:59.999
9
00:01:00,120 --> 00:01:11,370
connect them both to the data we might be able to use to shed these quantitative insights and to the goals,

99:59:59.999 --> 99:59:59.999
10
00:01:11,370 --> 00:01:17,870
the business purposes or scientific purposes for which we're asking the questions in the first place.

99:59:59.999 --> 99:59:59.999
11
00:01:17,870 --> 00:01:23,060
So I want to work through this with you with an example.

99:59:59.999 --> 99:59:59.999
12
00:01:23,060 --> 00:01:28,010
So suppose in the Boys State Computer Science Department, we have our introductory classes.

99:59:59.999 --> 99:59:59.999
13
00:01:28,010 --> 00:01:32,750
Yes. One twenty one to twenty one. Three twenty one. Suppose we make some change to see.

99:59:59.999 --> 99:59:59.999
14
00:01:32,750 --> 00:01:40,030
Yes. Twenty one. Like we change the way we do the assignments. And we want to assess whether this new change improved.

99:59:59.999 --> 99:59:59.999
15
00:01:40,030 --> 00:01:46,070
C. S 121. So we have a business purpose here of we're making a change to one of our courses.

99:59:59.999 --> 99:59:59.999
16
00:01:46,070 --> 00:01:50,540
And we want to see if that change is improving the course in some way.

99:59:59.999 --> 99:59:59.999
17
00:01:50,540 --> 00:01:56,580
But in order to do that, we need to identify a number of things, such as what does it mean to improve C.

99:59:59.999 --> 99:59:59.999
18
00:01:56,580 --> 00:02:04,510
S one twenty one? What data could we use to try to inform this assessment of whether we improved?

99:59:59.999 --> 99:59:59.999
19
00:02:04,510 --> 00:02:11,390
Yes, 121. And what could we do with that data to measure improvement?

99:59:59.999 --> 99:59:59.999
20
00:02:11,390 --> 00:02:17,540
And. So this process is called operationalization.

99:59:59.999 --> 99:59:59.999
21
00:02:17,540 --> 00:02:27,620
We have a goal. Assess whether we improved 121. That, in turn is in service of the broader goal of delivering a high quality undergraduate education.

99:59:59.999 --> 99:59:59.999
22
00:02:27,620 --> 00:02:34,760
Then we've refined through intermediate questions. I'm going to show some of those in a bit to determine a specific measurement to take.

99:59:59.999 --> 99:59:59.999
23
00:02:34,760 --> 00:02:39,470
And at the end of the day, if we are fully operationalize the goal or a question,

99:59:59.999 --> 99:59:59.999
24
00:02:39,470 --> 00:02:45,860
we know precisely what data we're going to collect or has been collected and how what measurement

99:59:59.999 --> 99:59:59.999
25
00:02:45,860 --> 00:02:51,140
or measurements we're going to compute over that data in order to try to answer our question,

99:59:59.999 --> 99:59:59.999
26
00:02:51,140 --> 00:02:55,940
we use the term in a couple of senses. First, operationalize can be a verb.

99:59:59.999 --> 99:59:59.999
27
00:02:55,940 --> 00:03:03,290
It's the process of doing this operationalization. Then, as a tense of the verb operationalization is also a noun.

99:59:59.999 --> 99:59:59.999
28
00:03:03,290 --> 00:03:05,330
And it's the result of this process.

99:59:59.999 --> 99:59:59.999
29
00:03:05,330 --> 00:03:15,500
So the specific measurement and analysis that we're going to do over specific data can be called an operationalization of the question.

99:59:59.999 --> 99:59:59.999
30
00:03:15,500 --> 00:03:19,760
So we have our goal of assessing whether some change improves.

99:59:59.999 --> 99:59:59.999
31
00:03:19,760 --> 00:03:24,290
Yes, 121. We can ask an intermediate question.

99:59:59.999 --> 99:59:59.999
32
00:03:24,290 --> 00:03:29,900
OK, so what does it mean to improve it? Well, students are better prepared to go excel in the workplace.

99:59:59.999 --> 99:59:59.999
33
00:03:29,900 --> 00:03:36,650
Well, it's a while until this is the freshman class. It's a while until the students are going out and the job market.

99:59:59.999 --> 99:59:59.999
34
00:03:36,650 --> 00:03:42,950
Or we have for information on how well equipped they were. So can we ask a shorter term question?

99:59:59.999 --> 99:59:59.999
35
00:03:42,950 --> 00:03:50,030
That's going to help us get to that. Are students better prepared for the next class?

99:59:59.999 --> 99:59:59.999
36
00:03:50,030 --> 00:03:57,100
And we call this intermediate question a proxy.

99:59:59.999 --> 99:59:59.999
37
00:03:57,100 --> 00:04:05,560
So if our goal is better, prepare them for doing their work, they're doing the work we're training them for the proxy can be.

99:59:59.999 --> 99:59:59.999
38
00:04:05,560 --> 00:04:09,280
Well, are they better prepared for the next class?

99:59:59.999 --> 99:59:59.999
39
00:04:09,280 --> 00:04:21,010
So questions don't have one level and there's a there's a path here between goal our goal, improve education, deliver a high quality education,

99:59:59.999 --> 99:59:59.999
40
00:04:21,010 --> 00:04:30,160
the subgoal of assess whether this change that was intended to improve the educational effectiveness of our introductory

99:59:59.999 --> 99:59:59.999
41
00:04:30,160 --> 00:04:38,620
programing class actually did so to get all the way down to the data that we can use in order to try to measure it.

99:59:59.999 --> 99:59:59.999
42
00:04:38,620 --> 00:04:43,300
We can also have multiple levels of questions, as we've already seen well.

99:59:59.999 --> 99:59:59.999
43
00:04:43,300 --> 00:04:47,170
Are they prepared for for their work? Well, we can't. That's a long timeframe.

99:59:59.999 --> 99:59:59.999
44
00:04:47,170 --> 00:04:53,650
It's difficult to measure that on the timeframe we need in order to iterate on the on class structures.

99:59:59.999 --> 99:59:59.999
45
00:04:53,650 --> 00:04:58,180
So we use this that we step down one level. We use this proxy.

99:59:59.999 --> 99:59:59.999
46
00:04:58,180 --> 00:05:01,630
Are they better prepared for the next class?

99:59:59.999 --> 99:59:59.999
47
00:05:01,630 --> 00:05:11,320
So if we want to think about the quality of our questions, like we need a way to assess whether a question is good.

99:59:59.999 --> 99:59:59.999
48
00:05:11,320 --> 00:05:14,210
And there's a couple of ways we do that. One is looking upward.

99:59:59.999 --> 99:59:59.999
49
00:05:14,210 --> 00:05:21,910
So the question should advance the goal and we should be able to look at the goal and look at the question and say yes.

99:59:59.999 --> 99:59:59.999
50
00:05:21,910 --> 00:05:26,950
Answering this question does move us forward in this goal.

99:59:59.999 --> 99:59:59.999
51
00:05:26,950 --> 00:05:30,760
No one question is going to be the complete answer to our goal.

99:59:59.999 --> 99:59:59.999
52
00:05:30,760 --> 00:05:35,710
But our students, better prepared for the next class, moves us one step closer.

99:59:59.999 --> 99:59:59.999
53
00:05:35,710 --> 00:05:40,300
We can say yes, if we if students are better prepared for the next class,

99:59:59.999 --> 99:59:59.999
54
00:05:40,300 --> 00:05:46,360
that is probably evidence that we have improved the effectiveness of the introductory class.

99:59:59.999 --> 99:59:59.999
55
00:05:46,360 --> 00:05:52,660
Also, though, carrying out the analysis should answer the question.

99:59:59.999 --> 99:59:59.999
56
00:05:52,660 --> 00:05:58,540
We want to work our questions down to the point where we have a question that's specific.

99:59:59.999 --> 99:59:59.999
57
00:05:58,540 --> 00:06:07,240
We we can it's clear that the question will advance either the top level goal or a higher level question that in turn advances the goal.

99:59:59.999 --> 99:59:59.999
58
00:06:07,240 --> 00:06:11,110
But also it's specific enough that we can look at a data analysis plan.

99:59:59.999 --> 99:59:59.999
59
00:06:11,110 --> 00:06:17,140
Here's the data we're going to use. Here's the measurements we're going to take. Here's the analysis we're going to perform and we can say, yes,

99:59:59.999 --> 99:59:59.999
60
00:06:17,140 --> 00:06:24,220
doing this data analysis plan will answer this question or at least answer the question in a useful sense.

99:59:59.999 --> 99:59:59.999
61
00:06:24,220 --> 00:06:31,240
And so if we can make those connections that we can see, doing the analysis will answer the question, answer the question.

99:59:59.999 --> 99:59:59.999
62
00:06:31,240 --> 00:06:35,350
Answering the question will advance the goal. Then we have a connection.

99:59:59.999 --> 99:59:59.999
63
00:06:35,350 --> 00:06:41,620
We have a connectedness between the analysis and the data that we can actually do.

99:59:59.999 --> 99:59:59.999
64
00:06:41,620 --> 00:06:48,720
And. The question or the goal that we're trying to advance through this data analysis.

99:59:59.999 --> 99:59:59.999
65
00:06:48,720 --> 00:06:57,030
So a fully operationalized question is going to be specific and it's going to be answerable and with the available data.

99:59:59.999 --> 99:59:59.999
66
00:06:57,030 --> 00:07:01,080
Now, there are lots of useful questions that we can't answer with available data.

99:59:59.999 --> 99:59:59.999
67
00:07:01,080 --> 00:07:11,080
That does not mean they're bad or we should ignore them. They're incredibly useful for contextualizing the limits of a data analysis that we do.

99:59:59.999 --> 99:59:59.999
68
00:07:11,080 --> 00:07:14,290
We have a data analysis. It can answer one question that will advance the goal.

99:59:59.999 --> 99:59:59.999
69
00:07:14,290 --> 00:07:18,760
There are three other questions related to the goal that cannot be answered by our analysis.

99:59:59.999 --> 99:59:59.999
70
00:07:18,760 --> 00:07:23,080
Well, that's useful in our report to talk about the limitations. Well, we can answer this question.

99:59:59.999 --> 99:59:59.999
71
00:07:23,080 --> 00:07:27,880
We can't answer these others. Maybe we can think about how to how to answer those others questions.

99:59:59.999 --> 99:59:59.999
72
00:07:27,880 --> 00:07:31,130
But when we're trying to get down to a question that we can answer with data.

99:59:59.999 --> 99:59:59.999
73
00:07:31,130 --> 00:07:39,370
And remember, we're talking about data sciences, quantitative insights into these questions.

99:59:59.999 --> 99:59:59.999
74
00:07:39,370 --> 00:07:45,580
We want to see, can we actually answer the question with data? And can we match the analysis plan to the question to the goal.

99:59:59.999 --> 99:59:59.999
75
00:07:45,580 --> 00:07:53,830
So to go back to our example of trying to measure the effectiveness of improving one twenty one, are students better prepared for the next class?

99:59:59.999 --> 99:59:59.999
76
00:07:53,830 --> 00:07:57,870
Well, we can make that more specific. Are they more likely to pass?

99:59:59.999 --> 99:59:59.999
77
00:07:57,870 --> 00:08:01,960
Yes. To twenty one. Now we have a very specific question.

99:59:59.999 --> 99:59:59.999
78
00:08:01,960 --> 00:08:07,930
We can answer it with the student grades from six to twenty one. We can look at students who took our class.

99:59:59.999 --> 99:59:59.999
79
00:08:07,930 --> 00:08:11,950
Our new C. S one twenty one and took our old C as one.

99:59:59.999 --> 99:59:59.999
80
00:08:11,950 --> 00:08:15,940
And we can compare the pass rates. Now there are many caveats.

99:59:59.999 --> 99:59:59.999
81
00:08:15,940 --> 00:08:21,640
There are a lot of challenges to doing this properly. It can only measure one piece of what's going on.

99:59:59.999 --> 99:59:59.999
82
00:08:21,640 --> 00:08:25,300
But it's a specific question that we can answer with data.

99:59:59.999 --> 99:59:59.999
83
00:08:25,300 --> 00:08:33,280
Our students in the new version of our intro class more or less likely to pass the next class,

99:59:59.999 --> 99:59:59.999
84
00:08:33,280 --> 00:08:37,090
will get to talk more about this question in the next video.

99:59:59.999 --> 99:59:59.999
85
00:08:37,090 --> 00:08:43,270
Now, to get to this kind of a question, I've given you the example and work through it here.

99:59:59.999 --> 99:59:59.999
86
00:08:43,270 --> 00:08:51,310
In practice, you're going to need to work with your boss, your client, your advisor, other stakeholders,

99:59:59.999 --> 99:59:59.999
87
00:08:51,310 --> 00:08:58,560
whoever is going to be acting on the results of your data analysis, which may be yourself.

99:59:59.999 --> 99:59:59.999
88
00:08:58,560 --> 00:09:05,190
To get to these operation, to get to these fully operationalized questions, they're going to have goals.

99:59:59.999 --> 99:59:59.999
89
00:09:05,190 --> 00:09:11,130
They may have some some high level questions, they may have some specific questions that can't map to the data.

99:59:59.999 --> 99:59:59.999
90
00:09:11,130 --> 00:09:17,370
One of the key ways to be able to do this refinement is through clarifying questions such as.

99:59:59.999 --> 99:59:59.999
91
00:09:17,370 --> 00:09:21,240
So if if the department chair came to you and said,

99:59:59.999 --> 99:59:59.999
92
00:09:21,240 --> 00:09:28,350
I would like you to help me measure the effect of this improvement to see us one twenty one, well, then we can ask questions.

99:59:59.999 --> 99:59:59.999
93
00:09:28,350 --> 00:09:34,520
What do we mean by improve? What would be evidence that we did improve?

99:59:59.999 --> 99:59:59.999
94
00:09:34,520 --> 00:09:39,500
Six one twenty one. And so we're gonna have practice in the synchronous time.

99:59:59.999 --> 99:59:59.999
95
00:09:39,500 --> 00:09:44,360
That's one of the things we're gonna do this week in thinking about clarifying questions.

99:59:59.999 --> 99:59:59.999
96
00:09:44,360 --> 00:09:48,050
But these clarifying questions that you can ask to your client.

99:59:59.999 --> 99:59:59.999
97
00:09:48,050 --> 00:09:52,310
We're going to use the term client generally for whoever is you're doing the data

99:59:59.999 --> 99:59:59.999
98
00:09:52,310 --> 00:09:57,670
analysis for to figure out what they actually want and what you can do with the data.

99:59:59.999 --> 99:59:59.999
99
00:09:57,670 --> 00:10:05,360
That's going to advance their goals. So to wrap up, there are multiple layers to translate between our high level goals,

99:59:59.999 --> 99:59:59.999
100
00:10:05,360 --> 00:10:11,030
deliver a high quality undergraduate education and what we can actually do with data

99:59:59.999 --> 99:59:59.999
101
00:10:11,030 --> 00:10:16,850
measure whether this change increased students ability to pass the next class.

99:59:59.999 --> 99:59:59.999
102
00:10:16,850 --> 00:10:26,570
Questions bridge this gap and we can have multiple layers of questions in order to get from high level goal to something we can do with data.

99:59:59.999 --> 99:59:59.999
103
00:10:26,570 --> 00:10:35,933
You're gonna be doing this a lot through the rest of the semester.

99:59:59.999 --> 99:59:59.999