Welcome back. And this video, I want to talk with you about what I'm going to call questioning questions I want us to dig deeper into.
We have a question. We have a candidate operationalization. How do we understand what it's actually going to be doing and how do we evaluate it?
So we're learning outcomes here are to be able to identify stakeholders and subgroups for a
problem and to evaluate the operationalization of a question to see who and what it prioritizes.
A key concept I want you to take away from this lecture is that we can understand
a metric or a measurement by asking what it takes to make it improve.
So to return to the example from the previous video, we want to assess whether a new change is improved.
Our introductory computer science class.
We can't one way we can go try to do that is to look at students who have taken their grades in the next class,
see us to twenty one and see if they're more likely to pass the next class.
We can measure this by looking at the grades. But this is not the only way that we can.
We can try to measure whether we've improved since to twenty or one twenty one.
So this option, I'm going to call it option one. We're looking at the pass rate. What fraction of students pass.
Yes. To twenty one for whatever definition of pass they get a C minus or better.
We could, we could up get. We could say what fraction get a B or get a B minus or better.
But what fraction of students pass. And does the Nusi as 121 method improve it.
Another way we could go try to look at it would be to look at the grades students receive in to twenty one.
So what is the average grade for students? For first students with the new intro.
With the previous intro and C two twenty one. Does the new, does the new technique in intro improve it.
We then have a question. Do we want to look at letter grades and, and maybe compute the average with the same formula used for GPA.
Do we want to look at coarse points. So actually look at what did they get in ninety five and ninety nine.
But we can look at the movement in the average grades now to evaluate what we have to figure out.
Which of these do we want to do. And a key tool that I want you to become familiar with for evaluating and understanding different measurements,
metrics, statistics, etc., is to ask how do I improve this measurement?
So we're computing's that. They were computing a pass rate, were computing an average grade.
How do I make this measurement better or worse? And this is going to give us a lot of insight into how the measurement works, what it prioritizes.
This this way of thinking about a measurement or a problem,
I think is going to serve you well throughout this class and throughout the rest of your education and work.
So let's talk about what these measurements and how to improve these measurements.
So if we want to improve the pass rate and we're not changing six to twenty one itself,
we can improve the pass rate by just passing everybody in to twenty one.
But if we're leaving to twenty one alone and we're changing, we're trying to improve the pass rate.
By making a change to the class that prepares students for it,
the only way to improve this is by helping those students who are going to have the most difficulty with C.
S one to one with C. S two one. If a student goes through one twenty one and they're going to get A, B and A and C.
S one twenty one. No change that we make to their excuse me to twenty one.
No change that we make to their one twenty one experience is going to improve our metric.
The only way to improve it is by helping more students move.
From a D to a C minus. Option two, why measuring the grade, we can improve this by making students do better at.
But we make do it by enabling students to do better.
If a student that would have gotten an A minus under the previous one twenty one is now prepared to the point where they'll get an A.
We improve it and we improve it just as much as if we move from a C minus to a C.
And so we there's more opportunity to make the grade better.
But we can improve this metric only by helping the students who were already well-prepared for C to twenty one.
So this brings us to a key point you get, which you measure.
If you set something up as the evaluation criteria, we're gonna see this really clearly once we start optimizing machine learning models.
When you set something up as your optimization criteria, that's what you get.
If you evaluate 120, what if you evaluate changes to the introductory class by people passing the next class, then you?
That's going to. That structures the pedagogy because design and the teaching evaluations to favor preparing students to pass the next class.
If you measure if you measure increased in average grade, that structure did to improve average grades.
But it might but it might focus the attention more on helping the students who were going to get a pretty good grade.
Anyway.
So going beyond, though, just just so we're looking here at two subgroups, one metric clearly favors the students who are on the edge pass rate.
You can only improve it by helping the students who are on the edge. The average grade, you can help across the board.
There are a variety of different stakeholders in the in the design of of an introductory class.
There's the students themselves who are going to be learning. There's the faculty who have to teach it.
Either they teach it themselves, they teach it directly, or they depend on students being prepared by that class as a prerequisite for the class.
They do teach the department, obviously, as a stakeholder because it has an interest in students producing a good education
that that students are want to come for and that employers want to hire for.
Employers want students to come out of the program well-prepared.
The university wants one has an interest in having programs that that produce well-prepared students.
And there also are attractive to students to be it to increase enrollment numbers.
But then even within within stake, the broad stakeholder categories, we have different subgroups.
So just within students, we can talk about high performing students.
We can talk about students who are underprepared for one or another class.
We can talk about students who are in some way or another marginalized.
And they may experience changes in the class structure of the class, delivery of the class assessment differently.
So even even once we've identified a stakeholder group, not every subset of that stakeholder group is going to experience.
What we're trying to study or is going to be reflected in the data in the same way,
we need to be able to identify these different groups to understand what it is that we're actually measuring.
So I want to return, though, this key question ask how do I improve this more?
Faced with a metric that really helps us clarify how a metric or a measurement behaves.
What changes to improve it? What also what can remain the same?
While it is improved. And then another important question is, how can it be gamed or manipulated?
Metrics are always what we call Lawsie and reductive.
Which means what that means is that no measurement captures everything about a phenomenon we care about.
And reductive means that we're we're taking a complex virts phenomenon.
We're reducing it down to one or a handful of measurements. We always lose something there.
But the results. But this question helps us evaluate what we're losing and iterate and improve our metrics and also assess and
assess whether those those weaknesses are actual challenges to validity or just something we need to keep in mind.
So to wrap up, it's crucial to appropriately define our measurements and then to question our definitions and a good import.
A useful way to do that is to ask how do we improve a metric that we're thinking about using?
And then also we want to look at who or what does a metric prioritize?
And is that prioritization is consistent with what we want to accomplish through organizational business or scientific goals.