This video, I'm going to introduce some of the fundamental structures and principles of doing scientific computing in Python.
Since the last couple of videos, I've briefly introduced Python's core structures and core data types.
But a lot of our work is going to be working with an additional set of structures,
a set of libraries known as scientific python or as the pie data stack.
So learning outcomes of this video are to understand limitations of core python data types for data science to know.
Three key rate data types particularly. Are we focusing primarily on the number high end the array?
Also briefly introduce serious and data frame. We're going to see a lot more about those next week and the.
To be able to perform basic vectorized operations. So in Python, we can write a list of numbers like this.
So numbers equals I'm using the list syntax that we talked about in the earlier video.
And I've got four numbers in here that I'm storing in this list and the variable numbers.
Now. This seems like a perfectly natural thing to do.
But remember, we said I said in the previous video that everything in Python is an object.
So this isn't just a list of numbers. If we wrote this in Java or C, we would have an array of numbers where system array.
And it's the stores, the numbers, one after the other. But in Python, that's not how it works because everything is an object.
What our list stores is, it stores pointers to numbers.
So we've got a list. And it's got a pointer to O point three and a pointer to nine point two, et cetera.
So what we store is the list itself has these pointers, which are eight bytes each.
And it has the. Numbers themselves.
A flooding point. A double precision flooding point number takes eight bites. But the numbers aren't just numbers, they're objects.
And every python object has at least 16 bites.
This is all on a 64 bit system, has at least 16 bytes of header information.
And so this whole list of numbers takes 144 bytes because we've the list has a header.
It has pointers. The pointers are the objects that have headers in addition to the data.
Also, the elements of a list can be different types. So when you go over the list, there's no guarantee that everything is a number.
So if we if we want to sum our numbers, there is a python function called some that will double do a sum.
But it's basically doing this. So we'll initialize a variable called total.
Well, then loop over all of our numbers and we'll add each one to the total.
And that's gonna make the total equal the total of the numbers. This works, it works just fine.
And for a list of four numbers, it's completely fine. But Python.
There's a couple of issues here. One python is Python. The language itself is rather slow.
It's quite convenient, but it's slow and it's slow for two reasons.
One is that it is interpreted the python code is compiled to an internal data structure,
but then there's C code that runs in a loop interpreting that data structure.
It's also dynamically typed. So remember, I said there's the the values and the numbers are in the list can have different types.
We wrote a set of numbers there. But Python isn't guaranteed that they're all numbers.
And so rather than saying, okay, I have a number, I'm going to keep adding it.
What it says is I have a thing and I'm going to try to add it to the thing I already have.
And it has to go look up how to do that, and it does that every time for each number.
This is all very slow. Also, since it's pointers, if you've taken the computer architecture class.
That may ring a few alarm bells for you because rather than just having an array of numbers which will be loaded into our cash very quickly accessed,
we have an array of pointers and each pointer has to go off and look up the number in memory.
And those numbers might be stored next to each other, but they might be stored all over the heap.
We're gonna have cash misses which make these slow process even slower so we can write code like this and it works fine,
but it's not an efficient way to do computation. And as we get to larger and larger data sets, you get a few hundred.
You got a few thousand numbers. You're gonna be fine. When you've got a million numbers, when you have a hundred million or a billion numbers.
Then things start to really get slow. So none PI is a python package that provides efficient data types for doing numeric computation.
And NUM Pi underlies almost all of the rest of the scientific python and data science and machine learning for Python software.
It has a data type called an NDA array. There's a variety of different ways you can create one,
but here we're going to just create one using the array constructor and then we're going to pass it our list.
So we're creating the list in this case. We are going to see later many ways to load arrays without having to go through a list.
I'm just doing this here so I can demonstrate how the array works.
But all the elements are of the same type in an array and they're also stored directly in the array.
So this ENDI array, it's the stores, the floats, one right after each other.
Eight bytes each. And so we don't have the indirection, three pointers. We don't have all of the overhead of storing all of these different objects.
It's just storing the numbers, one right after each other. You can have an Endi array of objects and that's going to store the pointers.
And that's useful in a few cases, especially for treating strings consistently with numbers.
But it really shines when we're dealing with arrays of numbers for various scientific computing applications.
So if you want to sum our numbers, we can use the num pi some function and it it's much shorter.
A little python has a some function, as I mentioned, that we could have used, but also it's implemented in a compiled language.
And when you have a num high array that's storing numbers, whether the integers,
whether they're floating point numbers, it's stored internally in a format that's compatible with C or Fortran.
And so a lot of num pi. Functions.
What they're doing is they're passing the array to see code or Fortran code or
C++ code that has a comp. loop that works on that data type and is able to very,
very efficiently sum up those numbers. We don't have a cast mate cash issues from the indirection.
We don't have the overhead of Python's interpreted code.
We don't have the overhead of having to deal with the the elements of the array might be of different types.
They're all the same type. We can work over them in in a loop, in comp.
Machine code. So in general, don't loop. You can loop over a number high end the array.
It's iterable just like a list. But in general, you don't want to do that.
You want to set up your code so that num pi can do the looping for you.
And effectively what we wind up using Python as is a scripting language to tell the underlying C, C++ and Fortran code.
What to do.
And the fact that Python is a slow language doesn't matter very much because the vast majority of our processing time won't be spent in Python.
So I thought none pile. So has a feature called Vector Ization.
There are a lot of operations that operate on an entire array at a time.
So if I get it, I can create another array. The Linn's base function here.
It. The land space function here.
It creates an array of four values that are evenly spaced from zero to one inclusive.
And then the plus operator here, remember, plus between two numbers is going to add it between two strings.
It's going to concatenate them plus between two arrays requires them to be of compatible shapes.
And it adds the the corresponding elements of the arrays to each other and returns a new array.
So what if we have a bunch of number one array of numbers and we have another array of numbers?
We want to add them together. We just add the two arrays and it does that addition again in a loop written in C or Fortran.
And it does it very, very quickly.
You can also add an integer or an integer or a floating point, single number to an array, and it'll add it to every element of the array.
But this is the key point to be able to make scientific computing with Python fast.
We setup our code and throughout. We're gonna be trying to set it up so that we use vectorized nation as much as possible.
And we vectorized over as much data at a time as possible so we can allow the optimized loops and in num pi,
in Pandas and Sai Pi and psychic learn to do the work and to put as much of the work as possible into those compile loops.
So we're not spending a lot of time in slow python code. Each array has three key things.
It has a data type called a D type, and that says what kind of elements are in the array?
PI has data types for your standard integers of various sizes.
Single and double precision floating point numbers. It also has D types for working with.
Date. Date. Times. Strings and then storing arrays.
That's where pointers to arbitrary python objects.
The data type or the array also has a shape which is a tuple of integers that says how big the array is.
The array may be multidimensional. So Endi array stands for N Dimensional Array.
And it can be one, two, three, four, whatever dimensional. So if we have a 100 by 50 matrix, it's stored in a in a number PI in the array of shape.
One hundred, comma 50. And then there's the data. It's stealth.
That's the elements of the array. The data points themselves that are stored in the array.
So then pandas, which we're going to see next week, builds on top of a raise with two new data types,
a series is an array with an associated index that allows us to look up.
So an ENDI array, like a python list is indexed using numbers starting from zero zero one, two, three, four, five.
But sometimes for a lot of times we're gonna have some other natural index. If you've taken databases, it's equivalent to the primary key.
So a series is an array with an associated index that might be other numbers.
That might be strings. But some other way of accessing the points.
It also has an efficient representations that you can have a series that's indexed zero through and minus one where N is the length of the series.
And it does not take up a lot of space to do that. And then a data frame is a table where each column is a series.
And they all share the same index. And we're gonna see those a lot because we load in a set of data points that's gonna be in a data frame.
Now, an assignment zero, you're going to briefly see both of these data structures.
I walk you through everything you have to do with them in assignment zero. And we're going to introduce them a lot more.
Woomera's talking about how to describe data next week.
But. Endi Arae, the number higher radiata structure is the fundamental core that all of these others are built on.
The series augments it with an index. The data frame collects multiple series together with column names like a spreadsheet table.
So we're still going to sometimes use Python native lists and loops.
Oftentimes, it's going to be because for some reason, we need a list of arrays or data frames.
Also, if we need to loop, if we have, say,
20 input files that we need to put together to to to be our data set or we got different groups of data, we're going to loop over those.
But the big thing we avoid doing is looping over individual data points.
We load in a few hundred thousand records. They're going to be in a data frame.
We don't loop over the rows of a data frame. If we can avoid it, because there's almost always a more efficient way to do that computation,
that pushes a lot of it into the C and C++ code and Fortran code that underlies NUM, Pi, pandas, et cetera.
So wrap up num pi provides efficient to ray data structures that are more memory compact's.
They don't take up nearly as much space and they're also much more efficient to compute over.
These are going to be the backbone of our data processing throughout the rest of the class.
And we want to prefer vectorized operations that perform these loops in native comp. machine code whenever possible for a little bit of practice.
I encourage you to take the example code from this from these slides and go and try them
in a notebook so you can get a little more practice creating notebooks and running code.
I will see you in class.