Blow in this video. I want to talk with you about basic operations for manipulating data,
learning outcomes for this video are for you to know key data reshaping operations and the corresponding Penders function.
Think about the process of transforming data in steps. This is a tour guide to the corresponding notebook.
I'm not showing the actual code in the video,
but you're going to see in the notebook how you actually implement different versions of each of these steps.
So we think about the shape of our data. We have rows and we have columns.
We have a certain number of rows and columns, each of a number, a type in a name.
The assumption we're going to make throughout these operations is that each row is another observation of the same kind of thing.
So our data are well organized. Each row we have the variables.
And that's gonna be one type of thing. So if you have a data frame of movies, each row represents one movie.
If we have data that's not in that kind of a format, we're going to talk about that later.
And sourcing and cleaning data. How do we get data in this kind of a tidy format?
For now, we're going to assume we have data in this format. This kind of a layout are the are Eco-System calls this tidy Vaida data.
Now, each of these methods return in new frame. A few of them are going to return a series.
But in general, we're gonna be transforming data frames to data frames here.
And so if our input is a data frame, each row is another observation of the same kind of thing.
The output will be a data frame. Each row is an observation of the same kind of thing.
It might be an observation, the same kind of thing as the input. It might be an observation of a different kind of thing.
But these are the different operations that we're going to be talking about here.
So if we want to select calls, we have a frame and we want the same frame.
But with fewer columns, we have few options. We can pick one column by treating the frame as a dictionary.
We can pick multiple columns bypassing in the list of column names to the same way we pick one column.
One column will yield a series. Multiple columns will yield a frame. If we want to remove a column.
So we want to keep all of the columns except one or two or however many that we name the drop method.
Ream returns a frame with all the columns of the original frame except the ones you tell it to drop.
We want to select rows. We have a frame. We want the same frame, but a subset of the rows.
A few common ways to do that or to select by a boolean mask. We set up a PAN, the series that has boolean values.
That's true. And all the for all the data positions we want to keep.
And then we select and then we. So this is really good if we want to select by column values.
So we can use a comparison operator to create a mask where all the values for one column are equal to a particular value.
And then we can select, we can select using that boolean mask.
We can select by position in the in the frame, starting with zero.
We can do that no matter what the index is with. I lock that is that's so lock is the location.
Accessor for Panda's data frames, I lock Access's by integer position, always lock indexes by index keys.
If we have the index keys we want, if we just load it. So we just loaded the data frame from a CSP file.
We haven't specified any index options. It's using the default range index.
Then selecting that position and index key are the same thing.
If we have a data frame with, call with, whereas we've got our observations and we've got color,
a column that identifies what some kind of a group that each observation is, then maybe it's ratings.
It's the movie. Maybe it's movies. And it's the actor, the genre.
And what we want is a frame or a series that has one row per group of the original data.
And it's computing some kind of a statistic from a value and all the rows, all of the rows within that group.
Then we want a group by an aggregate like we saw in the videos last week.
A couple more transformations are to think about tall versus wide data.
So why the data has a column per variable.
So in this case, if this is data, this is data of the of this average speed for each of four different stages of a cycling race.
And so we've got a column for each of the four different stages and end our rows or for each cyclist, total data.
Has its simplest form toll data or long data has three columns.
We have the road. We have the identifier. We have the variable name.
And we have the variable value. Sometimes this will just be called idee, variable and value.
But often it's often useful to give the variable and value name columns, meaningful names.
We could also have more than one idea. Call it if we need to.
But the idea here is that rather than having the stages in different columns, we split them out into a different row.
So cycle one cyclist one has four rows, one for each of the four stages.
We still call this an observation for one thing and for the same kind of thing.
It's just in the wide data. Each of our observations is for a cyclist and it's an observation of their speed for all four stages.
Whereas in the long data, each observation is for a cyclist.
One cyclist in one particular stage. So each cyclist will have four observations, one for each stage.
Total data is useful for plotting and grouping because a lot of our plotting function,
YouTube plotting utility functions are going to want to deal with a categorical variable that we use to determine maybe the x axis.
Maybe the color. And so often we're going to need tall data, especially when we're going to be plotting.
If you want to term why data in the tall, you use melt. If you want to turn tall data into why use the pivot with a pivot table methods and pandas.
You can also create tall data from a list. So if we have a data frame when one of the columns actually contains lists,
we haven't seen any data with this so far except the John Rós column in the Waj Movieland data.
But if we have a if we have a data frame or one column contains lists and we have and what we want is one row per list element.
So we want to take this list that's in a column and split it out so that each element gets another row where it's going to go ahead,
duplicate the rest of the column. So they're going to have their values repeated, whatever we're doing once for each of the elements.
This list, the pandas explode method. We'll do that. Then finally, to convert between series and data frame.
So if we have if we have a data frame and we want to get a series, we just select the column from the data.
We saw that the beginning. If we have a series and we want to get a data frame,
we can just create a single column frame with two frame and the two frame method on the serious object also.
But to give it a names that you have a name and the resulting data frame,
you can also if you want to create a multi column data frame where you've got a column for the value end,
you have a column for the index of the original of the original series.
The Pandas, the series Freeze Reset Index Method or pop that index out into a data frame column.
And then finally, if you have a series with multiple levels to its index, we haven't seen those yet, but we're going to see them from time to time.
The unstamped method will turn the inner most index labels in the column labels.
To turn it series into a data. So to think about strategy.
Each of these is an individual little building block. And we need to put them together to get from the data that we have to the data that we want.
And so what I recommend is that you decide what you want the end product to look like.
If you're going to draw a chart or you're going to do an analysis or an inference,
what are the observations and the variables that you need for that chart or inference?
And then once you've figured that out, you can plot a path from your current data to your end product.
So. If you want to show a distribution of the mean ratings for all of the movies in the horror genre, then you're going to need to select.
The rows that have the movies only that are in the horror genre, you can select that.
You're probably going to have a join as well in order to get the genre table and the movie table connected, depending on how your data's laid out.
And once you've filtered it down, OK, these are the horror movies that you need to get the ratings and you need to you need to be able.
You need to get there.
You need to have the average ratings, you need to combine those with the movies, as we've seen the ability to do in a previous video.
And then you have the observations that you want. You need to be able to plot this kind of a path and what you have at the end product.
In this example, I've reference some Joynes. We saw joints very, very briefly last week.
We're going to see them again in more detail in the notebooks. So to wrap up, Penders has many tools for reshaping data.
You want to start with the end in mind, work from what you have to what you need.
Read the tutorial notebooks for a lot more details.