So this video, we're want to talk about asking questions.
What makes a good question? How does a question relate to the broader context of what we're trying to do in this class?
The learning outcomes for this video are few to understand what makes a good question.
Understand how it relates to goals and analysis and start to think about data for a question.
We're also going to introduce a key term operationalization that is going to come up throughout the rest of the class.
To set the stage, I want to review our definition of data science that I introduced in the class introduction video that we're learning
about how to use data to provide quantitative insights on questions of scientific business or social interest.
But in order to do that effectively, we need to be able to write good questions, refine those questions,
connect them both to the data we might be able to use to shed these quantitative insights and to the goals,
the business purposes or scientific purposes for which we're asking the questions in the first place.
So I want to work through this with you with an example.
So suppose in the Boys State Computer Science Department, we have our introductory classes.
Yes. One twenty one to twenty one. Three twenty one. Suppose we make some change to see.
Yes. Twenty one. Like we change the way we do the assignments. And we want to assess whether this new change improved.
C. S 121. So we have a business purpose here of we're making a change to one of our courses.
And we want to see if that change is improving the course in some way.
But in order to do that, we need to identify a number of things, such as what does it mean to improve C.
S one twenty one? What data could we use to try to inform this assessment of whether we improved?
Yes, 121. And what could we do with that data to measure improvement?
And. So this process is called operationalization.
We have a goal. Assess whether we improved 121. That, in turn is in service of the broader goal of delivering a high quality undergraduate education.
Then we've refined through intermediate questions. I'm going to show some of those in a bit to determine a specific measurement to take.
And at the end of the day, if we are fully operationalize the goal or a question,
we know precisely what data we're going to collect or has been collected and how what measurement
or measurements we're going to compute over that data in order to try to answer our question,
we use the term in a couple of senses. First, operationalize can be a verb.
It's the process of doing this operationalization. Then, as a tense of the verb operationalization is also a noun.
And it's the result of this process.
So the specific measurement and analysis that we're going to do over specific data can be called an operationalization of the question.
So we have our goal of assessing whether some change improves.
Yes, 121. We can ask an intermediate question.
OK, so what does it mean to improve it? Well, students are better prepared to go excel in the workplace.
Well, it's a while until this is the freshman class. It's a while until the students are going out and the job market.
Or we have for information on how well equipped they were. So can we ask a shorter term question?
That's going to help us get to that. Are students better prepared for the next class?
And we call this intermediate question a proxy.
So if our goal is better, prepare them for doing their work, they're doing the work we're training them for the proxy can be.
Well, are they better prepared for the next class?
So questions don't have one level and there's a there's a path here between goal our goal, improve education, deliver a high quality education,
the subgoal of assess whether this change that was intended to improve the educational effectiveness of our introductory
programing class actually did so to get all the way down to the data that we can use in order to try to measure it.
We can also have multiple levels of questions, as we've already seen well.
Are they prepared for for their work? Well, we can't. That's a long timeframe.
It's difficult to measure that on the timeframe we need in order to iterate on the on class structures.
So we use this that we step down one level. We use this proxy.
Are they better prepared for the next class?
So if we want to think about the quality of our questions, like we need a way to assess whether a question is good.
And there's a couple of ways we do that. One is looking upward.
So the question should advance the goal and we should be able to look at the goal and look at the question and say yes.
Answering this question does move us forward in this goal.
No one question is going to be the complete answer to our goal.
But our students, better prepared for the next class, moves us one step closer.
We can say yes, if we if students are better prepared for the next class,
that is probably evidence that we have improved the effectiveness of the introductory class.
Also, though, carrying out the analysis should answer the question.
We want to work our questions down to the point where we have a question that's specific.
We we can it's clear that the question will advance either the top level goal or a higher level question that in turn advances the goal.
But also it's specific enough that we can look at a data analysis plan.
Here's the data we're going to use. Here's the measurements we're going to take. Here's the analysis we're going to perform and we can say, yes,
doing this data analysis plan will answer this question or at least answer the question in a useful sense.
And so if we can make those connections that we can see, doing the analysis will answer the question, answer the question.
Answering the question will advance the goal. Then we have a connection.
We have a connectedness between the analysis and the data that we can actually do.
And. The question or the goal that we're trying to advance through this data analysis.
So a fully operationalized question is going to be specific and it's going to be answerable and with the available data.
Now, there are lots of useful questions that we can't answer with available data.
That does not mean they're bad or we should ignore them. They're incredibly useful for contextualizing the limits of a data analysis that we do.
We have a data analysis. It can answer one question that will advance the goal.
There are three other questions related to the goal that cannot be answered by our analysis.
Well, that's useful in our report to talk about the limitations. Well, we can answer this question.
We can't answer these others. Maybe we can think about how to how to answer those others questions.
But when we're trying to get down to a question that we can answer with data.
And remember, we're talking about data sciences, quantitative insights into these questions.
We want to see, can we actually answer the question with data? And can we match the analysis plan to the question to the goal.
So to go back to our example of trying to measure the effectiveness of improving one twenty one, are students better prepared for the next class?
Well, we can make that more specific. Are they more likely to pass?
Yes. To twenty one. Now we have a very specific question.
We can answer it with the student grades from six to twenty one. We can look at students who took our class.
Our new C. S one twenty one and took our old C as one.
And we can compare the pass rates. Now there are many caveats.
There are a lot of challenges to doing this properly. It can only measure one piece of what's going on.
But it's a specific question that we can answer with data.
Our students in the new version of our intro class more or less likely to pass the next class,
will get to talk more about this question in the next video.
Now, to get to this kind of a question, I've given you the example and work through it here.
In practice, you're going to need to work with your boss, your client, your advisor, other stakeholders,
whoever is going to be acting on the results of your data analysis, which may be yourself.
To get to these operation, to get to these fully operationalized questions, they're going to have goals.
They may have some some high level questions, they may have some specific questions that can't map to the data.
One of the key ways to be able to do this refinement is through clarifying questions such as.
So if if the department chair came to you and said,
I would like you to help me measure the effect of this improvement to see us one twenty one, well, then we can ask questions.
What do we mean by improve? What would be evidence that we did improve?
Six one twenty one. And so we're gonna have practice in the synchronous time.
That's one of the things we're gonna do this week in thinking about clarifying questions.
But these clarifying questions that you can ask to your client.
We're going to use the term client generally for whoever is you're doing the data
analysis for to figure out what they actually want and what you can do with the data.
That's going to advance their goals. So to wrap up, there are multiple layers to translate between our high level goals,
deliver a high quality undergraduate education and what we can actually do with data
measure whether this change increased students ability to pass the next class.
Questions bridge this gap and we can have multiple layers of questions in order to get from high level goal to something we can do with data.
You're gonna be doing this a lot through the rest of the semester.