https:/.../80e425d4-f295-4f1b-b73a-ad9601830f06-917f40b9-6e6e-4fdb-92ac-ad9c006eabef.mp4?invocationId=3832aaf4-a60f-ec11-a9e9-0a1a827ad0ec

1
00:00:08,280 --> 00:00:13,300
Hello again. And this video, I want to talk about organizing notebooks as I've promised.

2
00:00:13,300 --> 00:00:18,210
So we've talked about how do we make charts? That's been a lot of what we've been talking about here.

3
00:00:18,210 --> 00:00:25,260
But I wanted to talk about how do we actually put together a notebook that's presenting these charts and presenting our conclusions from them.

4
00:00:25,260 --> 00:00:33,040
So learning outcomes for this video are for you to be able to use markdown document structure to organize a notebook, to use the Jupiter,

5
00:00:33,040 --> 00:00:40,350
a markdown features to format text in a notebook to create a notebook that clearly tells the story of a data analysis.

6
00:00:40,350 --> 00:00:47,610
First thing to understand is that a notebook is a document. It is a convenient way to run Python code and to see the results of it.

7
00:00:47,610 --> 00:00:52,950
But the notebook structure is first and foremost a document. It's meant to be read.

8
00:00:52,950 --> 00:01:00,300
And there's some structure imposed in the document because it has to read in the same order as the code is going to execute.

9
00:01:00,300 --> 00:01:08,420
But. We want to be able to actually read it and understand what's going on as we walk through the notebook.

10
00:01:08,420 --> 00:01:12,830
So we also want to factor particularly complex computations out of the notebook.

11
00:01:12,830 --> 00:01:15,410
So far, nothing. We've been doing a super complex.

12
00:01:15,410 --> 00:01:22,220
But if I have a large, complicated data processing operation, training and an extensive set of machine learning models or something,

13
00:01:22,220 --> 00:01:33,200
I'll put those out of the notebook and other scripts and other modules and leave the notebook for communicating the results of my data analysis.

14
00:01:33,200 --> 00:01:38,570
So a notebook has two primary types of cells. We have code cells, which you've seen a lot.

15
00:01:38,570 --> 00:01:44,720
The Python code and its output. And we have marked down cells that contain formatted text.

16
00:01:44,720 --> 00:01:48,260
Could keep you. I recommend keeping your code cells relatively short.

17
00:01:48,260 --> 00:01:57,050
One, a few lines. One function definition. If you're defining an entire class and it's taking 100 lines within a code cell.

18
00:01:57,050 --> 00:02:01,580
That's a good sign you to pull that out into a python module of some kind.

19
00:02:01,580 --> 00:02:06,440
If helpful, show results after the cell. I do this a lot, particularly in development.

20
00:02:06,440 --> 00:02:11,450
But if you have too much of it, it can make it hard to read the final notebook because you have all of these outputs.

21
00:02:11,450 --> 00:02:15,440
And the notebook wins it being a sea of charts and tables.

22
00:02:15,440 --> 00:02:20,600
And it's difficult to find your way through the notebook and find the pieces that you need to look at.

23
00:02:20,600 --> 00:02:27,020
So go ahead. Do a lot of them, especially while you're debugging in your prototyping before you submit.

24
00:02:27,020 --> 00:02:32,750
Maybe go through and clean up, remove things that were just there for you to test how something worked and leave the cells in

25
00:02:32,750 --> 00:02:37,940
your notebook being the ones that help the reader understand the results of what you're doing.

26
00:02:37,940 --> 00:02:45,790
Remember, the purpose of the presentation is to show the reader what you learned and how you know it's true.

27
00:02:45,790 --> 00:02:54,040
Cells that didn't help you do that. Maybe you can consider removing, though, or that don't help you do that.

28
00:02:54,040 --> 00:02:58,720
At the end of the day, they might have helped you figure out how to do that.

29
00:02:58,720 --> 00:03:02,650
You can save up a copy of your notebook before doing the cleanup so you don't lose them.

30
00:03:02,650 --> 00:03:07,210
You can have a supplementary notebook that has maybe Pazz, you went down.

31
00:03:07,210 --> 00:03:11,170
That didn't work out. Another thing you can consider doing is having an appendix.

32
00:03:11,170 --> 00:03:17,020
So you've got all of the main content, the notebook. And then down at the end, you have a big heading appendix.

33
00:03:17,020 --> 00:03:20,770
And there you have extra things. You want to make sure you can still run from top to bottom.

34
00:03:20,770 --> 00:03:27,490
But there you have some of the other things that maybe dove into more details about the building blocks of some of your computations.

35
00:03:27,490 --> 00:03:32,650
But it is good to show the results after loading data and after doing a complex manipulation,

36
00:03:32,650 --> 00:03:38,910
especially one that significantly changes the shape of the data that you're working with.

37
00:03:38,910 --> 00:03:41,790
And they talk mostly in this video, though, about markdown sales,

38
00:03:41,790 --> 00:03:48,180
because markdown sells or what you use to build up the structure of your document and make it tell a story,

39
00:03:48,180 --> 00:03:52,830
not just be a kind of strange way to present Python code.

40
00:03:52,830 --> 00:04:01,710
So markdown is a text syntax for simple markup. I'm going to provide a link to the markdown documentation in the class notes that go with this video.

41
00:04:01,710 --> 00:04:06,570
But there's several inline formatting things. If you put two stars around some text, that'll make it bold.

42
00:04:06,570 --> 00:04:16,110
One star will make it italics. You can indicate a code using the fit, something that's going to show up as the fixed width code layout using back Tex.

43
00:04:16,110 --> 00:04:24,570
This is one that I see ignored very frequently in and writing up because it's really,

44
00:04:24,570 --> 00:04:27,960
really useful for function names, variable names, things like that.

45
00:04:27,960 --> 00:04:32,670
To be able to set apart like this is a special thing. This is a function name also.

46
00:04:32,670 --> 00:04:39,180
Then you can use tech math syntax by putting it between dollar signs in this markdown notebook.

47
00:04:39,180 --> 00:04:48,450
Pay attention to the details of what your markdown code or what your text formatted text looks like after you render it in the notebook.

48
00:04:48,450 --> 00:04:55,500
Make sure it reads well. Make sure it's clear. Ask yourself if I weren't the one who right wrote this.

49
00:04:55,500 --> 00:05:03,290
What? I like reading this and clean it up and pay attention to those details, to make it look,

50
00:05:03,290 --> 00:05:08,670
to make it look good and to make it be effective at communicating and so that the reader

51
00:05:08,670 --> 00:05:13,170
can clearly understand what the different pieces are and what needs to be emphasized,

52
00:05:13,170 --> 00:05:17,260
etc. Markdown also has a number of block elements.

53
00:05:17,260 --> 00:05:21,520
The basic one is a paragraph, paragraphs or just text separated by blank lines.

54
00:05:21,520 --> 00:05:26,890
You can also have bulleted and numbered lists. You can have code blocks for if you need to have a little.

55
00:05:26,890 --> 00:05:31,720
These aren't super common in a notebook because a lot of your code is in the code cells that you execute.

56
00:05:31,720 --> 00:05:34,900
But if you need to have a little code that you don't execute for some reason,

57
00:05:34,900 --> 00:05:42,970
you can put it in the code block and markdown and then you can also block mathematics, a line on its own that begins and ends with two dollar signs.

58
00:05:42,970 --> 00:05:49,170
And you can actually span multiple lines so long as there aren't any blanks that's going to be treated as a piece of block mathematics.

59
00:05:49,170 --> 00:05:54,510
It's not in line in a sentence, but it becomes its own block and the rendered self.

60
00:05:54,510 --> 00:06:00,450
Headings are an important one to pay attention to. So Mark Down headings are lines that start with one, two,

61
00:06:00,450 --> 00:06:06,510
three up to six hash marks and then a space in the heading text having one heading to hitting three.

62
00:06:06,510 --> 00:06:11,370
Something that's important to know is the hashes do not mean big and bold.

63
00:06:11,370 --> 00:06:16,890
That's what they look like. But that's not what they mean. What they mean is heading.

64
00:06:16,890 --> 00:06:23,300
And so you need to have an outline structure to your notebook using the headings.

65
00:06:23,300 --> 00:06:29,030
And you need to nest them properly, so within each one, you have your H 2s.

66
00:06:29,030 --> 00:06:32,760
And then you have your H threes. You don't go straight from H one to H for you.

67
00:06:32,760 --> 00:06:41,540
You have H three in the middle. Start the notebook with an H one that has the notebook title and that that will become in a lot of rendering context.

68
00:06:41,540 --> 00:06:46,850
That becomes the title at the top of your notebook. And then all your other headings are two or lower.

69
00:06:46,850 --> 00:06:55,670
Also you might if you have an appendix, you might have Appendix B, another H1, but also the section headers should be short, not sentences.

70
00:06:55,670 --> 00:07:00,630
If you're writing an entire sentence in your section header. You're you're putting too much there.

71
00:07:00,630 --> 00:07:06,840
The section header should be a short title and then the section content comes after it.

72
00:07:06,840 --> 00:07:13,650
Now, one of the few reasons why it's important to use the section headers heading levels properly.

73
00:07:13,650 --> 00:07:19,710
One is just visually, it helps break up your notebook so we can easily see which component we're at.

74
00:07:19,710 --> 00:07:26,130
Second, there are extensions that will do things like no your headings or give you a Browsr Bowl table of contents.

75
00:07:26,130 --> 00:07:32,130
You can use to navigate the notebook by heading what? I'm rendering notebooks as a part of the course website.

76
00:07:32,130 --> 00:07:36,960
You'll see this over in the right hand side. You can jump directly to notebook headings.

77
00:07:36,960 --> 00:07:45,720
That only works because I'm consistently using the heading levels to build the structure and outline based structure of my notebook.

78
00:07:45,720 --> 00:07:53,160
Another a third reason is for accessibility. If someone's reading your notebook with an assistive technology such as a screen reader,

79
00:07:53,160 --> 00:08:01,500
the section headings are very important to help them navigate to the parts. The notebook that are both relevant to them at a given time.

80
00:08:01,500 --> 00:08:09,180
So on the section headers, one additional little rule is if your section editor has to wrap onto a second line, really rethink.

81
00:08:09,180 --> 00:08:13,230
It's almost certainly too long particularly.

82
00:08:13,230 --> 00:08:16,860
Don't put an entire question in the section. Maybe. Usually.

83
00:08:16,860 --> 00:08:22,890
Occasionally it's OK to put a whole question, but maybe put a brief like three to five word summary of the questions topic and

84
00:08:22,890 --> 00:08:30,750
then write this question itself as the first paragraph of the of the section.

85
00:08:30,750 --> 00:08:33,240
But pay attention to these different formatting features.

86
00:08:33,240 --> 00:08:41,460
You can build a well-structured notebook that communicates clearly and draws the reader's emphasis to the places where it needs to go.

87
00:08:41,460 --> 00:08:46,650
Writing the text itself. Use the document to tell a story. What's the goal of what you're doing?

88
00:08:46,650 --> 00:08:51,630
Either the whole notebook or of individual pieces of analysis. What's the data that we're doing?

89
00:08:51,630 --> 00:08:58,140
What do we know about it going in at the up at the top, either at the very top of your notebook or where you're loading the data?

90
00:08:58,140 --> 00:09:04,050
It's useful to write some, especially at the notebooks and we report you submit to somebody.

91
00:09:04,050 --> 00:09:07,800
It's useful to write there. What do you know? Where did you get the data? How was it collected?

92
00:09:07,800 --> 00:09:16,290
Not a full data sheet, but at least some summary information to help the reader understand what it is that we're going to be going and looking at.

93
00:09:16,290 --> 00:09:19,800
Why are we doing each piece of the analysis? What's the purpose here?

94
00:09:19,800 --> 00:09:25,260
How does it fit into our broader picture, into our broader goals? What approach are we using?

95
00:09:25,260 --> 00:09:28,200
We don't want to just repeat the code writing a a numbered list here.

96
00:09:28,200 --> 00:09:33,240
The steps and those steps are just a literal translation of the code doesn't help understanding.

97
00:09:33,240 --> 00:09:37,570
It creates an opportunity for a code and documentation to become mismatched.

98
00:09:37,570 --> 00:09:41,670
But explain if there's anything tricky in the code.

99
00:09:41,670 --> 00:09:48,600
Explain why that does the job. Explain the conceptual idea behind why you're approaching things the way you are.

100
00:09:48,600 --> 00:09:56,700
If you're doing a data clean up, explain why that what that cleanup's doing and why that's the right cleanup for your data.

101
00:09:56,700 --> 00:10:03,600
And then what do we learn from it? So oftentimes what I do with us, with an individual piece of it, like a chart.

102
00:10:03,600 --> 00:10:07,260
All right. What question the charge is supposed to be answering.

103
00:10:07,260 --> 00:10:11,010
Or at least the purpose of the chart that we have the code to generate the chart itself.

104
00:10:11,010 --> 00:10:18,450
And then we have a tech cell that has observations about what we learn from the chart.

105
00:10:18,450 --> 00:10:23,160
So what are we doing? How are we going to do it if that's not immediately clear code results?

106
00:10:23,160 --> 00:10:25,590
And then what do we observe from these results?

107
00:10:25,590 --> 00:10:33,350
So the over then the high level document structure that I recommend is to start start with the title and intros.

108
00:10:33,350 --> 00:10:37,140
You've got your title. You're heading one. Then what's the notebook for?

109
00:10:37,140 --> 00:10:41,160
Why does this notebook exist? Are there to include links?

110
00:10:41,160 --> 00:10:46,650
There's hyperlinks and taxes and markdown as well. Read the markdown documentation to see how to use that.

111
00:10:46,650 --> 00:10:50,790
But where does this go? Are there things we need to know?

112
00:10:50,790 --> 00:10:54,990
Background about where why this documents being created?

113
00:10:54,990 --> 00:11:00,360
Where did the data come from? If we have defined research questions, what are those research questions?

114
00:11:00,360 --> 00:11:03,750
You can write those right in the intro, the notebook. Then I have a set up.

115
00:11:03,750 --> 00:11:09,060
I almost always have a setup section that comes next. That has input. I import my python libraries.

116
00:11:09,060 --> 00:11:11,970
I've maybe defined some help or functions that I'm going to be using throughout the

117
00:11:11,970 --> 00:11:16,860
notebook helper function specific to one section I might define in that section.

118
00:11:16,860 --> 00:11:20,730
But then and then I LOEs load the data. Sometimes I load the data as a part of the setup.

119
00:11:20,730 --> 00:11:27,990
So it's OK. Important modules and then load my data. Sometimes if specially if I have more to say about the data, it's its own section.

120
00:11:27,990 --> 00:11:34,500
But then as I load each table, I just show the first few rows of it often so that I can see, OK, I've loaded this data and then it's right there.

121
00:11:34,500 --> 00:11:39,390
We can see as we're going through the rest of the notebook. What is the data just loaded look like?

122
00:11:39,390 --> 00:11:45,030
Then we perform our analysis and this might be two sections. It might be five, six, seven, eight sections.

123
00:11:45,030 --> 00:11:49,920
And then finally at the end, we can summarize and conclude this is going to be really I don't always do this in

124
00:11:49,920 --> 00:11:53,340
my research notebook because often that's the material that goes in the paper.

125
00:11:53,340 --> 00:11:57,150
But this is going to be something particularly in our assignment, and we're submitting notebooks.

126
00:11:57,150 --> 00:12:00,810
Put that at the end of the notebook. What do we learn from this?

127
00:12:00,810 --> 00:12:03,150
Sometimes they going to have specific directions for things.

128
00:12:03,150 --> 00:12:08,730
I want you to reflect on there when like an assignment one, I've broken down the different requirements.

129
00:12:08,730 --> 00:12:15,120
Those become good candidates for your age to your level. Two headings for each of those.

130
00:12:15,120 --> 00:12:21,730
So we've got, I think six require six different requirements. An assignment one.

131
00:12:21,730 --> 00:12:28,120
H2, heading a primary section of your document for each of those is a good starting point for your layout.

132
00:12:28,120 --> 00:12:34,460
In addition to you're probably gonna have another one up at the top for the setup and maybe another for the data load.

133
00:12:34,460 --> 00:12:38,590
But think about this. This the flow, your document be able to communicate.

134
00:12:38,590 --> 00:12:42,760
What are we doing? What are the prerequisites in terms of and data?

135
00:12:42,760 --> 00:12:47,790
How are we actually doing it? And then at the end, what do we learn?

136
00:12:47,790 --> 00:12:53,580
So you're going to write a lot of cells and produce a lot of outputs in your notebook while you're debugging,

137
00:12:53,580 --> 00:12:59,220
before you submit to before you share in other contexts. Spend some time cleaning up your notebook,

138
00:12:59,220 --> 00:13:06,780
remove dead ends and extraneous outputs that you included for debugging, but don't fit in the flow of the story.

139
00:13:06,780 --> 00:13:13,350
Consider putting them in a supplementary notebook. If you want to keep them around and then make sure you can rerun your notebook from top to bottom.

140
00:13:13,350 --> 00:13:20,300
So when the Jupiter interface is the kernel, when you click that and choose, restart and rerun all.

141
00:13:20,300 --> 00:13:26,570
And it will restart the python kernel that's actually running your code so all your variables disappear.

142
00:13:26,570 --> 00:13:31,220
Your data is no longer loaded. And then it starts running the notebook from top to bottom.

143
00:13:31,220 --> 00:13:37,220
You want that to succeed so that someone else working with the notebook can actually rerun and reproduce your results.

144
00:13:37,220 --> 00:13:43,790
If that doesn't succeed, then that means either you deleted something that's that's important or you're the order of

145
00:13:43,790 --> 00:13:48,740
your source in the notebook does not match the order in which it actually has to be executed.

146
00:13:48,740 --> 00:13:54,980
But make sure it succeeds and also read back to the notebook to make sure that the charts all still look right.

147
00:13:54,980 --> 00:14:01,100
The data is the conclusions are all still correct, etc. before you submit the final notebook.

148
00:14:01,100 --> 00:14:05,330
So when you're writing an up of two, you also need to know your audience and your purpose.

149
00:14:05,330 --> 00:14:10,550
For example, the notebooks I'm writing for you for teaching purposes here.

150
00:14:10,550 --> 00:14:17,060
They the things I write in them differ from what I'm going to write in a research notebook that I share with my collaborators,

151
00:14:17,060 --> 00:14:22,700
or I use my own purposes because my purpose partially in their notebooks, is to explain how they're working.

152
00:14:22,700 --> 00:14:28,850
So I'm going to say more in these notebooks about how exactly the what exactly the code is

153
00:14:28,850 --> 00:14:35,040
doing is that you can learn how the code works that I would expect in a research notebook.

154
00:14:35,040 --> 00:14:41,730
But also, you're your own internal your own personal use sharing with your adviser or your supervisor,

155
00:14:41,730 --> 00:14:47,970
sharing with the public, either the professional public working on your topic or the general public.

156
00:14:47,970 --> 00:14:54,180
These are all different audiences and they're going to need different levels of explanation and different things highlighted in your notebook.

157
00:14:54,180 --> 00:14:56,820
Also, not all audiences are well served for notebooks.

158
00:14:56,820 --> 00:15:04,500
Notebooks are fantastic for internal reports, collaboration, et cetera, sharing the results of a data analysis with colleagues or with yourself.

159
00:15:04,500 --> 00:15:09,090
But for final publication, you're often going to need a separate final report.

160
00:15:09,090 --> 00:15:15,930
I don't know that it's possible to write a research paper and Jupiter notebooks. Somebody might have tried, but.

161
00:15:15,930 --> 00:15:21,180
But I'll still have the notebook where I explain the analysis. I often make that notebook available.

162
00:15:21,180 --> 00:15:31,050
So for a lot of my a lot of my published research papers, you can download a zip file or a get repository that contains the notebooks and you

163
00:15:31,050 --> 00:15:36,240
can rerun the experiment and rerun my analysis with the notebooks in the notebook.

164
00:15:36,240 --> 00:15:40,370
Then also I write the files out to disk. And we're not going to see this quite yet.

165
00:15:40,370 --> 00:15:45,630
We're going to see it later when we start talking about workflow. Because right now I'm just having to submit notebooks.

166
00:15:45,630 --> 00:15:49,720
But the note, the figures as they show up in the notebook aren't very high resolution.

167
00:15:49,720 --> 00:15:55,200
So we're gonna want to render a higher resolution version of them to a PMG file or a PDA file or a

168
00:15:55,200 --> 00:16:01,110
postscript file that we can then include in our document and word or law tech or whatever we're writing.

169
00:16:01,110 --> 00:16:08,910
So to wrap up, your notebook is first and foremost a document that contains code to generate the results that you're trying to discuss.

170
00:16:08,910 --> 00:16:14,490
Take advantage of the document structure and use it as a store to tell the story of your analysis.

171
00:16:14,490 --> 00:16:19,910
The conclusion you come to in why we should believe them. Pay attention to the examples I'm giving you in class.

172
00:16:19,910 --> 00:16:36,043
I'm also going to be trying to give you some examples of research oriented notebooks that you can look at to see examples of good notebook practice.