Hello again. And this video, I want to talk about organizing notebooks as I've promised.
So we've talked about how do we make charts? That's been a lot of what we've been talking about here.
But I wanted to talk about how do we actually put together a notebook that's presenting these charts and presenting our conclusions from them.
So learning outcomes for this video are for you to be able to use markdown document structure to organize a notebook, to use the Jupiter,
a markdown features to format text in a notebook to create a notebook that clearly tells the story of a data analysis.
First thing to understand is that a notebook is a document. It is a convenient way to run Python code and to see the results of it.
But the notebook structure is first and foremost a document. It's meant to be read.
And there's some structure imposed in the document because it has to read in the same order as the code is going to execute.
But. We want to be able to actually read it and understand what's going on as we walk through the notebook.
So we also want to factor particularly complex computations out of the notebook.
So far, nothing. We've been doing a super complex.
But if I have a large, complicated data processing operation, training and an extensive set of machine learning models or something,
I'll put those out of the notebook and other scripts and other modules and leave the notebook for communicating the results of my data analysis.
So a notebook has two primary types of cells. We have code cells, which you've seen a lot.
The Python code and its output. And we have marked down cells that contain formatted text.
Could keep you. I recommend keeping your code cells relatively short.
One, a few lines. One function definition. If you're defining an entire class and it's taking 100 lines within a code cell.
That's a good sign you to pull that out into a python module of some kind.
If helpful, show results after the cell. I do this a lot, particularly in development.
But if you have too much of it, it can make it hard to read the final notebook because you have all of these outputs.
And the notebook wins it being a sea of charts and tables.
And it's difficult to find your way through the notebook and find the pieces that you need to look at.
So go ahead. Do a lot of them, especially while you're debugging in your prototyping before you submit.
Maybe go through and clean up, remove things that were just there for you to test how something worked and leave the cells in
your notebook being the ones that help the reader understand the results of what you're doing.
Remember, the purpose of the presentation is to show the reader what you learned and how you know it's true.
Cells that didn't help you do that. Maybe you can consider removing, though, or that don't help you do that.
At the end of the day, they might have helped you figure out how to do that.
You can save up a copy of your notebook before doing the cleanup so you don't lose them.
You can have a supplementary notebook that has maybe Pazz, you went down.
That didn't work out. Another thing you can consider doing is having an appendix.
So you've got all of the main content, the notebook. And then down at the end, you have a big heading appendix.
And there you have extra things. You want to make sure you can still run from top to bottom.
But there you have some of the other things that maybe dove into more details about the building blocks of some of your computations.
But it is good to show the results after loading data and after doing a complex manipulation,
especially one that significantly changes the shape of the data that you're working with.
And they talk mostly in this video, though, about markdown sales,
because markdown sells or what you use to build up the structure of your document and make it tell a story,
not just be a kind of strange way to present Python code.
So markdown is a text syntax for simple markup. I'm going to provide a link to the markdown documentation in the class notes that go with this video.
But there's several inline formatting things. If you put two stars around some text, that'll make it bold.
One star will make it italics. You can indicate a code using the fit, something that's going to show up as the fixed width code layout using back Tex.
This is one that I see ignored very frequently in and writing up because it's really,
really useful for function names, variable names, things like that.
To be able to set apart like this is a special thing. This is a function name also.
Then you can use tech math syntax by putting it between dollar signs in this markdown notebook.
Pay attention to the details of what your markdown code or what your text formatted text looks like after you render it in the notebook.
Make sure it reads well. Make sure it's clear. Ask yourself if I weren't the one who right wrote this.
What? I like reading this and clean it up and pay attention to those details, to make it look,
to make it look good and to make it be effective at communicating and so that the reader
can clearly understand what the different pieces are and what needs to be emphasized,
etc. Markdown also has a number of block elements.
The basic one is a paragraph, paragraphs or just text separated by blank lines.
You can also have bulleted and numbered lists. You can have code blocks for if you need to have a little.
These aren't super common in a notebook because a lot of your code is in the code cells that you execute.
But if you need to have a little code that you don't execute for some reason,
you can put it in the code block and markdown and then you can also block mathematics, a line on its own that begins and ends with two dollar signs.
And you can actually span multiple lines so long as there aren't any blanks that's going to be treated as a piece of block mathematics.
It's not in line in a sentence, but it becomes its own block and the rendered self.
Headings are an important one to pay attention to. So Mark Down headings are lines that start with one, two,
three up to six hash marks and then a space in the heading text having one heading to hitting three.
Something that's important to know is the hashes do not mean big and bold.
That's what they look like. But that's not what they mean. What they mean is heading.
And so you need to have an outline structure to your notebook using the headings.
And you need to nest them properly, so within each one, you have your H 2s.
And then you have your H threes. You don't go straight from H one to H for you.
You have H three in the middle. Start the notebook with an H one that has the notebook title and that that will become in a lot of rendering context.
That becomes the title at the top of your notebook. And then all your other headings are two or lower.
Also you might if you have an appendix, you might have Appendix B, another H1, but also the section headers should be short, not sentences.
If you're writing an entire sentence in your section header. You're you're putting too much there.
The section header should be a short title and then the section content comes after it.
Now, one of the few reasons why it's important to use the section headers heading levels properly.
One is just visually, it helps break up your notebook so we can easily see which component we're at.
Second, there are extensions that will do things like no your headings or give you a Browsr Bowl table of contents.
You can use to navigate the notebook by heading what? I'm rendering notebooks as a part of the course website.
You'll see this over in the right hand side. You can jump directly to notebook headings.
That only works because I'm consistently using the heading levels to build the structure and outline based structure of my notebook.
Another a third reason is for accessibility. If someone's reading your notebook with an assistive technology such as a screen reader,
the section headings are very important to help them navigate to the parts. The notebook that are both relevant to them at a given time.
So on the section headers, one additional little rule is if your section editor has to wrap onto a second line, really rethink.
It's almost certainly too long particularly.
Don't put an entire question in the section. Maybe. Usually.
Occasionally it's OK to put a whole question, but maybe put a brief like three to five word summary of the questions topic and
then write this question itself as the first paragraph of the of the section.
But pay attention to these different formatting features.
You can build a well-structured notebook that communicates clearly and draws the reader's emphasis to the places where it needs to go.
Writing the text itself. Use the document to tell a story. What's the goal of what you're doing?
Either the whole notebook or of individual pieces of analysis. What's the data that we're doing?
What do we know about it going in at the up at the top, either at the very top of your notebook or where you're loading the data?
It's useful to write some, especially at the notebooks and we report you submit to somebody.
It's useful to write there. What do you know? Where did you get the data? How was it collected?
Not a full data sheet, but at least some summary information to help the reader understand what it is that we're going to be going and looking at.
Why are we doing each piece of the analysis? What's the purpose here?
How does it fit into our broader picture, into our broader goals? What approach are we using?
We don't want to just repeat the code writing a a numbered list here.
The steps and those steps are just a literal translation of the code doesn't help understanding.
It creates an opportunity for a code and documentation to become mismatched.
But explain if there's anything tricky in the code.
Explain why that does the job. Explain the conceptual idea behind why you're approaching things the way you are.
If you're doing a data clean up, explain why that what that cleanup's doing and why that's the right cleanup for your data.
And then what do we learn from it? So oftentimes what I do with us, with an individual piece of it, like a chart.
All right. What question the charge is supposed to be answering.
Or at least the purpose of the chart that we have the code to generate the chart itself.
And then we have a tech cell that has observations about what we learn from the chart.
So what are we doing? How are we going to do it if that's not immediately clear code results?
And then what do we observe from these results?
So the over then the high level document structure that I recommend is to start start with the title and intros.
You've got your title. You're heading one. Then what's the notebook for?
Why does this notebook exist? Are there to include links?
There's hyperlinks and taxes and markdown as well. Read the markdown documentation to see how to use that.
But where does this go? Are there things we need to know?
Background about where why this documents being created?
Where did the data come from? If we have defined research questions, what are those research questions?
You can write those right in the intro, the notebook. Then I have a set up.
I almost always have a setup section that comes next. That has input. I import my python libraries.
I've maybe defined some help or functions that I'm going to be using throughout the
notebook helper function specific to one section I might define in that section.
But then and then I LOEs load the data. Sometimes I load the data as a part of the setup.
So it's OK. Important modules and then load my data. Sometimes if specially if I have more to say about the data, it's its own section.
But then as I load each table, I just show the first few rows of it often so that I can see, OK, I've loaded this data and then it's right there.
We can see as we're going through the rest of the notebook. What is the data just loaded look like?
Then we perform our analysis and this might be two sections. It might be five, six, seven, eight sections.
And then finally at the end, we can summarize and conclude this is going to be really I don't always do this in
my research notebook because often that's the material that goes in the paper.
But this is going to be something particularly in our assignment, and we're submitting notebooks.
Put that at the end of the notebook. What do we learn from this?
Sometimes they going to have specific directions for things.
I want you to reflect on there when like an assignment one, I've broken down the different requirements.
Those become good candidates for your age to your level. Two headings for each of those.
So we've got, I think six require six different requirements. An assignment one.
H2, heading a primary section of your document for each of those is a good starting point for your layout.
In addition to you're probably gonna have another one up at the top for the setup and maybe another for the data load.
But think about this. This the flow, your document be able to communicate.
What are we doing? What are the prerequisites in terms of and data?
How are we actually doing it? And then at the end, what do we learn?
So you're going to write a lot of cells and produce a lot of outputs in your notebook while you're debugging,
before you submit to before you share in other contexts. Spend some time cleaning up your notebook,
remove dead ends and extraneous outputs that you included for debugging, but don't fit in the flow of the story.
Consider putting them in a supplementary notebook. If you want to keep them around and then make sure you can rerun your notebook from top to bottom.
So when the Jupiter interface is the kernel, when you click that and choose, restart and rerun all.
And it will restart the python kernel that's actually running your code so all your variables disappear.
Your data is no longer loaded. And then it starts running the notebook from top to bottom.
You want that to succeed so that someone else working with the notebook can actually rerun and reproduce your results.
If that doesn't succeed, then that means either you deleted something that's that's important or you're the order of
your source in the notebook does not match the order in which it actually has to be executed.
But make sure it succeeds and also read back to the notebook to make sure that the charts all still look right.
The data is the conclusions are all still correct, etc. before you submit the final notebook.
So when you're writing an up of two, you also need to know your audience and your purpose.
For example, the notebooks I'm writing for you for teaching purposes here.
They the things I write in them differ from what I'm going to write in a research notebook that I share with my collaborators,
or I use my own purposes because my purpose partially in their notebooks, is to explain how they're working.
So I'm going to say more in these notebooks about how exactly the what exactly the code is
doing is that you can learn how the code works that I would expect in a research notebook.
But also, you're your own internal your own personal use sharing with your adviser or your supervisor,
sharing with the public, either the professional public working on your topic or the general public.
These are all different audiences and they're going to need different levels of explanation and different things highlighted in your notebook.
Also, not all audiences are well served for notebooks.
Notebooks are fantastic for internal reports, collaboration, et cetera, sharing the results of a data analysis with colleagues or with yourself.
But for final publication, you're often going to need a separate final report.
I don't know that it's possible to write a research paper and Jupiter notebooks. Somebody might have tried, but.
But I'll still have the notebook where I explain the analysis. I often make that notebook available.
So for a lot of my a lot of my published research papers, you can download a zip file or a get repository that contains the notebooks and you
can rerun the experiment and rerun my analysis with the notebooks in the notebook.
Then also I write the files out to disk. And we're not going to see this quite yet.
We're going to see it later when we start talking about workflow. Because right now I'm just having to submit notebooks.
But the note, the figures as they show up in the notebook aren't very high resolution.
So we're gonna want to render a higher resolution version of them to a PMG file or a PDA file or a
postscript file that we can then include in our document and word or law tech or whatever we're writing.
So to wrap up, your notebook is first and foremost a document that contains code to generate the results that you're trying to discuss.
Take advantage of the document structure and use it as a store to tell the story of your analysis.
The conclusion you come to in why we should believe them. Pay attention to the examples I'm giving you in class.
I'm also going to be trying to give you some examples of research oriented notebooks that you can look at to see examples of good notebook practice.