1
99:59:59,999 --> 99:59:59,999
1
00:00:04,190 --> 00:00:09,250
So this video, we're want to talk about asking questions.

2
99:59:59,999 --> 99:59:59,999
2
00:00:09,250 --> 00:00:16,020
What makes a good question? How does a question relate to the broader context of what we're trying to do in this class?

3
99:59:59,999 --> 99:59:59,999
3
00:00:16,020 --> 00:00:21,130
The learning outcomes for this video are few to understand what makes a good question.

4
99:59:59,999 --> 99:59:59,999
4
00:00:21,130 --> 00:00:26,470
Understand how it relates to goals and analysis and start to think about data for a question.

5
99:59:59,999 --> 99:59:59,999
5
00:00:26,470 --> 00:00:35,790
We're also going to introduce a key term operationalization that is going to come up throughout the rest of the class.

6
99:59:59,999 --> 99:59:59,999
6
00:00:35,790 --> 00:00:44,400
To set the stage, I want to review our definition of data science that I introduced in the class introduction video that we're learning

7
99:59:59,999 --> 99:59:59,999
7
00:00:44,400 --> 00:00:52,500
about how to use data to provide quantitative insights on questions of scientific business or social interest.

8
99:59:59,999 --> 99:59:59,999
8
00:00:52,500 --> 00:01:00,120
But in order to do that effectively, we need to be able to write good questions, refine those questions,

9
99:59:59,999 --> 99:59:59,999
9
00:01:00,120 --> 00:01:11,370
connect them both to the data we might be able to use to shed these quantitative insights and to the goals,

10
99:59:59,999 --> 99:59:59,999
10
00:01:11,370 --> 00:01:17,870
the business purposes or scientific purposes for which we're asking the questions in the first place.

11
99:59:59,999 --> 99:59:59,999
11
00:01:17,870 --> 00:01:23,060
So I want to work through this with you with an example.

12
99:59:59,999 --> 99:59:59,999
12
00:01:23,060 --> 00:01:28,010
So suppose in the Boys State Computer Science Department, we have our introductory classes.

13
99:59:59,999 --> 99:59:59,999
13
00:01:28,010 --> 00:01:32,750
Yes. One twenty one to twenty one. Three twenty one. Suppose we make some change to see.

14
99:59:59,999 --> 99:59:59,999
14
00:01:32,750 --> 00:01:40,030
Yes. Twenty one. Like we change the way we do the assignments. And we want to assess whether this new change improved.

15
99:59:59,999 --> 99:59:59,999
15
00:01:40,030 --> 00:01:46,070
C. S 121. So we have a business purpose here of we're making a change to one of our courses.

16
99:59:59,999 --> 99:59:59,999
16
00:01:46,070 --> 00:01:50,540
And we want to see if that change is improving the course in some way.

17
99:59:59,999 --> 99:59:59,999
17
00:01:50,540 --> 00:01:56,580
But in order to do that, we need to identify a number of things, such as what does it mean to improve C.

18
99:59:59,999 --> 99:59:59,999
18
00:01:56,580 --> 00:02:04,510
S one twenty one? What data could we use to try to inform this assessment of whether we improved?

19
99:59:59,999 --> 99:59:59,999
19
00:02:04,510 --> 00:02:11,390
Yes, 121. And what could we do with that data to measure improvement?

20
99:59:59,999 --> 99:59:59,999
20
00:02:11,390 --> 00:02:17,540
And. So this process is called operationalization.

21
99:59:59,999 --> 99:59:59,999
21
00:02:17,540 --> 00:02:27,620
We have a goal. Assess whether we improved 121. That, in turn is in service of the broader goal of delivering a high quality undergraduate education.

22
99:59:59,999 --> 99:59:59,999
22
00:02:27,620 --> 00:02:34,760
Then we've refined through intermediate questions. I'm going to show some of those in a bit to determine a specific measurement to take.

23
99:59:59,999 --> 99:59:59,999
23
00:02:34,760 --> 00:02:39,470
And at the end of the day, if we are fully operationalize the goal or a question,

24
99:59:59,999 --> 99:59:59,999
24
00:02:39,470 --> 00:02:45,860
we know precisely what data we're going to collect or has been collected and how what measurement

25
99:59:59,999 --> 99:59:59,999
25
00:02:45,860 --> 00:02:51,140
or measurements we're going to compute over that data in order to try to answer our question,

26
99:59:59,999 --> 99:59:59,999
26
00:02:51,140 --> 00:02:55,940
we use the term in a couple of senses. First, operationalize can be a verb.

27
99:59:59,999 --> 99:59:59,999
27
00:02:55,940 --> 00:03:03,290
It's the process of doing this operationalization. Then, as a tense of the verb operationalization is also a noun.

28
99:59:59,999 --> 99:59:59,999
28
00:03:03,290 --> 00:03:05,330
And it's the result of this process.

29
99:59:59,999 --> 99:59:59,999
29
00:03:05,330 --> 00:03:15,500
So the specific measurement and analysis that we're going to do over specific data can be called an operationalization of the question.

30
99:59:59,999 --> 99:59:59,999
30
00:03:15,500 --> 00:03:19,760
So we have our goal of assessing whether some change improves.

31
99:59:59,999 --> 99:59:59,999
31
00:03:19,760 --> 00:03:24,290
Yes, 121. We can ask an intermediate question.

32
99:59:59,999 --> 99:59:59,999
32
00:03:24,290 --> 00:03:29,900
OK, so what does it mean to improve it? Well, students are better prepared to go excel in the workplace.

33
99:59:59,999 --> 99:59:59,999
33
00:03:29,900 --> 00:03:36,650
Well, it's a while until this is the freshman class. It's a while until the students are going out and the job market.

34
99:59:59,999 --> 99:59:59,999
34
00:03:36,650 --> 00:03:42,950
Or we have for information on how well equipped they were. So can we ask a shorter term question?

35
99:59:59,999 --> 99:59:59,999
35
00:03:42,950 --> 00:03:50,030
That's going to help us get to that. Are students better prepared for the next class?

36
99:59:59,999 --> 99:59:59,999
36
00:03:50,030 --> 00:03:57,100
And we call this intermediate question a proxy.

37
99:59:59,999 --> 99:59:59,999
37
00:03:57,100 --> 00:04:05,560
So if our goal is better, prepare them for doing their work, they're doing the work we're training them for the proxy can be.

38
99:59:59,999 --> 99:59:59,999
38
00:04:05,560 --> 00:04:09,280
Well, are they better prepared for the next class?

39
99:59:59,999 --> 99:59:59,999
39
00:04:09,280 --> 00:04:21,010
So questions don't have one level and there's a there's a path here between goal our goal, improve education, deliver a high quality education,

40
99:59:59,999 --> 99:59:59,999
40
00:04:21,010 --> 00:04:30,160
the subgoal of assess whether this change that was intended to improve the educational effectiveness of our introductory

41
99:59:59,999 --> 99:59:59,999
41
00:04:30,160 --> 00:04:38,620
programing class actually did so to get all the way down to the data that we can use in order to try to measure it.

42
99:59:59,999 --> 99:59:59,999
42
00:04:38,620 --> 00:04:43,300
We can also have multiple levels of questions, as we've already seen well.

43
99:59:59,999 --> 99:59:59,999
43
00:04:43,300 --> 00:04:47,170
Are they prepared for for their work? Well, we can't. That's a long timeframe.

44
99:59:59,999 --> 99:59:59,999
44
00:04:47,170 --> 00:04:53,650
It's difficult to measure that on the timeframe we need in order to iterate on the on class structures.

45
99:59:59,999 --> 99:59:59,999
45
00:04:53,650 --> 00:04:58,180
So we use this that we step down one level. We use this proxy.

46
99:59:59,999 --> 99:59:59,999
46
00:04:58,180 --> 00:05:01,630
Are they better prepared for the next class?

47
99:59:59,999 --> 99:59:59,999
47
00:05:01,630 --> 00:05:11,320
So if we want to think about the quality of our questions, like we need a way to assess whether a question is good.

48
99:59:59,999 --> 99:59:59,999
48
00:05:11,320 --> 00:05:14,210
And there's a couple of ways we do that. One is looking upward.

49
99:59:59,999 --> 99:59:59,999
49
00:05:14,210 --> 00:05:21,910
So the question should advance the goal and we should be able to look at the goal and look at the question and say yes.

50
99:59:59,999 --> 99:59:59,999
50
00:05:21,910 --> 00:05:26,950
Answering this question does move us forward in this goal.

51
99:59:59,999 --> 99:59:59,999
51
00:05:26,950 --> 00:05:30,760
No one question is going to be the complete answer to our goal.

52
99:59:59,999 --> 99:59:59,999
52
00:05:30,760 --> 00:05:35,710
But our students, better prepared for the next class, moves us one step closer.

53
99:59:59,999 --> 99:59:59,999
53
00:05:35,710 --> 00:05:40,300
We can say yes, if we if students are better prepared for the next class,

54
99:59:59,999 --> 99:59:59,999
54
00:05:40,300 --> 00:05:46,360
that is probably evidence that we have improved the effectiveness of the introductory class.

55
99:59:59,999 --> 99:59:59,999
55
00:05:46,360 --> 00:05:52,660
Also, though, carrying out the analysis should answer the question.

56
99:59:59,999 --> 99:59:59,999
56
00:05:52,660 --> 00:05:58,540
We want to work our questions down to the point where we have a question that's specific.

57
99:59:59,999 --> 99:59:59,999
57
00:05:58,540 --> 00:06:07,240
We we can it's clear that the question will advance either the top level goal or a higher level question that in turn advances the goal.

58
99:59:59,999 --> 99:59:59,999
58
00:06:07,240 --> 00:06:11,110
But also it's specific enough that we can look at a data analysis plan.

59
99:59:59,999 --> 99:59:59,999
59
00:06:11,110 --> 00:06:17,140
Here's the data we're going to use. Here's the measurements we're going to take. Here's the analysis we're going to perform and we can say, yes,

60
99:59:59,999 --> 99:59:59,999
60
00:06:17,140 --> 00:06:24,220
doing this data analysis plan will answer this question or at least answer the question in a useful sense.

61
99:59:59,999 --> 99:59:59,999
61
00:06:24,220 --> 00:06:31,240
And so if we can make those connections that we can see, doing the analysis will answer the question, answer the question.

62
99:59:59,999 --> 99:59:59,999
62
00:06:31,240 --> 00:06:35,350
Answering the question will advance the goal. Then we have a connection.

63
99:59:59,999 --> 99:59:59,999
63
00:06:35,350 --> 00:06:41,620
We have a connectedness between the analysis and the data that we can actually do.

64
99:59:59,999 --> 99:59:59,999
64
00:06:41,620 --> 00:06:48,720
And. The question or the goal that we're trying to advance through this data analysis.

65
99:59:59,999 --> 99:59:59,999
65
00:06:48,720 --> 00:06:57,030
So a fully operationalized question is going to be specific and it's going to be answerable and with the available data.

66
99:59:59,999 --> 99:59:59,999
66
00:06:57,030 --> 00:07:01,080
Now, there are lots of useful questions that we can't answer with available data.

67
99:59:59,999 --> 99:59:59,999
67
00:07:01,080 --> 00:07:11,080
That does not mean they're bad or we should ignore them. They're incredibly useful for contextualizing the limits of a data analysis that we do.

68
99:59:59,999 --> 99:59:59,999
68
00:07:11,080 --> 00:07:14,290
We have a data analysis. It can answer one question that will advance the goal.

69
99:59:59,999 --> 99:59:59,999
69
00:07:14,290 --> 00:07:18,760
There are three other questions related to the goal that cannot be answered by our analysis.

70
99:59:59,999 --> 99:59:59,999
70
00:07:18,760 --> 00:07:23,080
Well, that's useful in our report to talk about the limitations. Well, we can answer this question.

71
99:59:59,999 --> 99:59:59,999
71
00:07:23,080 --> 00:07:27,880
We can't answer these others. Maybe we can think about how to how to answer those others questions.

72
99:59:59,999 --> 99:59:59,999
72
00:07:27,880 --> 00:07:31,130
But when we're trying to get down to a question that we can answer with data.

73
99:59:59,999 --> 99:59:59,999
73
00:07:31,130 --> 00:07:39,370
And remember, we're talking about data sciences, quantitative insights into these questions.

74
99:59:59,999 --> 99:59:59,999
74
00:07:39,370 --> 00:07:45,580
We want to see, can we actually answer the question with data? And can we match the analysis plan to the question to the goal.

75
99:59:59,999 --> 99:59:59,999
75
00:07:45,580 --> 00:07:53,830
So to go back to our example of trying to measure the effectiveness of improving one twenty one, are students better prepared for the next class?

76
99:59:59,999 --> 99:59:59,999
76
00:07:53,830 --> 00:07:57,870
Well, we can make that more specific. Are they more likely to pass?

77
99:59:59,999 --> 99:59:59,999
77
00:07:57,870 --> 00:08:01,960
Yes. To twenty one. Now we have a very specific question.

78
99:59:59,999 --> 99:59:59,999
78
00:08:01,960 --> 00:08:07,930
We can answer it with the student grades from six to twenty one. We can look at students who took our class.

79
99:59:59,999 --> 99:59:59,999
79
00:08:07,930 --> 00:08:11,950
Our new C. S one twenty one and took our old C as one.

80
99:59:59,999 --> 99:59:59,999
80
00:08:11,950 --> 00:08:15,940
And we can compare the pass rates. Now there are many caveats.

81
99:59:59,999 --> 99:59:59,999
81
00:08:15,940 --> 00:08:21,640
There are a lot of challenges to doing this properly. It can only measure one piece of what's going on.

82
99:59:59,999 --> 99:59:59,999
82
00:08:21,640 --> 00:08:25,300
But it's a specific question that we can answer with data.

83
99:59:59,999 --> 99:59:59,999
83
00:08:25,300 --> 00:08:33,280
Our students in the new version of our intro class more or less likely to pass the next class,

84
99:59:59,999 --> 99:59:59,999
84
00:08:33,280 --> 00:08:37,090
will get to talk more about this question in the next video.

85
99:59:59,999 --> 99:59:59,999
85
00:08:37,090 --> 00:08:43,270
Now, to get to this kind of a question, I've given you the example and work through it here.

86
99:59:59,999 --> 99:59:59,999
86
00:08:43,270 --> 00:08:51,310
In practice, you're going to need to work with your boss, your client, your advisor, other stakeholders,

87
99:59:59,999 --> 99:59:59,999
87
00:08:51,310 --> 00:08:58,560
whoever is going to be acting on the results of your data analysis, which may be yourself.

88
99:59:59,999 --> 99:59:59,999
88
00:08:58,560 --> 00:09:05,190
To get to these operation, to get to these fully operationalized questions, they're going to have goals.

89
99:59:59,999 --> 99:59:59,999
89
00:09:05,190 --> 00:09:11,130
They may have some some high level questions, they may have some specific questions that can't map to the data.

90
99:59:59,999 --> 99:59:59,999
90
00:09:11,130 --> 00:09:17,370
One of the key ways to be able to do this refinement is through clarifying questions such as.

91
99:59:59,999 --> 99:59:59,999
91
00:09:17,370 --> 00:09:21,240
So if if the department chair came to you and said,

92
99:59:59,999 --> 99:59:59,999
92
00:09:21,240 --> 00:09:28,350
I would like you to help me measure the effect of this improvement to see us one twenty one, well, then we can ask questions.

93
99:59:59,999 --> 99:59:59,999
93
00:09:28,350 --> 00:09:34,520
What do we mean by improve? What would be evidence that we did improve?

94
99:59:59,999 --> 99:59:59,999
94
00:09:34,520 --> 00:09:39,500
Six one twenty one. And so we're gonna have practice in the synchronous time.

95
99:59:59,999 --> 99:59:59,999
95
00:09:39,500 --> 00:09:44,360
That's one of the things we're gonna do this week in thinking about clarifying questions.

96
99:59:59,999 --> 99:59:59,999
96
00:09:44,360 --> 00:09:48,050
But these clarifying questions that you can ask to your client.

97
99:59:59,999 --> 99:59:59,999
97
00:09:48,050 --> 00:09:52,310
We're going to use the term client generally for whoever is you're doing the data

98
99:59:59,999 --> 99:59:59,999
98
00:09:52,310 --> 00:09:57,670
analysis for to figure out what they actually want and what you can do with the data.

99
99:59:59,999 --> 99:59:59,999
99
00:09:57,670 --> 00:10:05,360
That's going to advance their goals. So to wrap up, there are multiple layers to translate between our high level goals,

100
99:59:59,999 --> 99:59:59,999
100
00:10:05,360 --> 00:10:11,030
deliver a high quality undergraduate education and what we can actually do with data

101
99:59:59,999 --> 99:59:59,999
101
00:10:11,030 --> 00:10:16,850
measure whether this change increased students ability to pass the next class.

102
99:59:59,999 --> 99:59:59,999
102
00:10:16,850 --> 00:10:26,570
Questions bridge this gap and we can have multiple layers of questions in order to get from high level goal to something we can do with data.

103
99:59:59,999 --> 99:59:59,999
103
00:10:26,570 --> 00:10:35,933
You're gonna be doing this a lot through the rest of the semester.

104
99:59:59,999 --> 99:59:59,999