[Script Info]
Title: 
[Events]
Format: Layer, Start, End, Style, Name, MarginL, MarginR, MarginV, Effect, Text
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,1\N00:00:08,280 --> 00:00:13,300\NHello again. And this video, I want to talk about organizing notebooks as I've promised.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,2\N00:00:13,300 --> 00:00:18,210\NSo we've talked about how do we make charts? That's been a lot of what we've been talking about here.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,3\N00:00:18,210 --> 00:00:25,260\NBut I wanted to talk about how do we actually put together a notebook that's presenting these charts and presenting our conclusions from them.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,4\N00:00:25,260 --> 00:00:33,040\NSo learning outcomes for this video are for you to be able to use markdown document structure to organize a notebook, to use the Jupiter,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,5\N00:00:33,040 --> 00:00:40,350\Na markdown features to format text in a notebook to create a notebook that clearly tells the story of a data analysis.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,6\N00:00:40,350 --> 00:00:47,610\NFirst thing to understand is that a notebook is a document. It is a convenient way to run Python code and to see the results of it.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,7\N00:00:47,610 --> 00:00:52,950\NBut the notebook structure is first and foremost a document. It's meant to be read.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,8\N00:00:52,950 --> 00:01:00,300\NAnd there's some structure imposed in the document because it has to read in the same order as the code is going to execute.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,9\N00:01:00,300 --> 00:01:08,420\NBut. We want to be able to actually read it and understand what's going on as we walk through the notebook.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,10\N00:01:08,420 --> 00:01:12,830\NSo we also want to factor particularly complex computations out of the notebook.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,11\N00:01:12,830 --> 00:01:15,410\NSo far, nothing. We've been doing a super complex.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,12\N00:01:15,410 --> 00:01:22,220\NBut if I have a large, complicated data processing operation, training and an extensive set of machine learning models or something,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,13\N00:01:22,220 --> 00:01:33,200\NI'll put those out of the notebook and other scripts and other modules and leave the notebook for communicating the results of my data analysis.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,14\N00:01:33,200 --> 00:01:38,570\NSo a notebook has two primary types of cells. We have code cells, which you've seen a lot.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,15\N00:01:38,570 --> 00:01:44,720\NThe Python code and its output. And we have marked down cells that contain formatted text.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,16\N00:01:44,720 --> 00:01:48,260\NCould keep you. I recommend keeping your code cells relatively short.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,17\N00:01:48,260 --> 00:01:57,050\NOne, a few lines. One function definition. If you're defining an entire class and it's taking 100 lines within a code cell.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,18\N00:01:57,050 --> 00:02:01,580\NThat's a good sign you to pull that out into a python module of some kind.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,19\N00:02:01,580 --> 00:02:06,440\NIf helpful, show results after the cell. I do this a lot, particularly in development.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,20\N00:02:06,440 --> 00:02:11,450\NBut if you have too much of it, it can make it hard to read the final notebook because you have all of these outputs.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,21\N00:02:11,450 --> 00:02:15,440\NAnd the notebook wins it being a sea of charts and tables.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,22\N00:02:15,440 --> 00:02:20,600\NAnd it's difficult to find your way through the notebook and find the pieces that you need to look at.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,23\N00:02:20,600 --> 00:02:27,020\NSo go ahead. Do a lot of them, especially while you're debugging in your prototyping before you submit.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,24\N00:02:27,020 --> 00:02:32,750\NMaybe go through and clean up, remove things that were just there for you to test how something worked and leave the cells in
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,25\N00:02:32,750 --> 00:02:37,940\Nyour notebook being the ones that help the reader understand the results of what you're doing.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,26\N00:02:37,940 --> 00:02:45,790\NRemember, the purpose of the presentation is to show the reader what you learned and how you know it's true.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,27\N00:02:45,790 --> 00:02:54,040\NCells that didn't help you do that. Maybe you can consider removing, though, or that don't help you do that.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,28\N00:02:54,040 --> 00:02:58,720\NAt the end of the day, they might have helped you figure out how to do that.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,29\N00:02:58,720 --> 00:03:02,650\NYou can save up a copy of your notebook before doing the cleanup so you don't lose them.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,30\N00:03:02,650 --> 00:03:07,210\NYou can have a supplementary notebook that has maybe Pazz, you went down.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,31\N00:03:07,210 --> 00:03:11,170\NThat didn't work out. Another thing you can consider doing is having an appendix.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,32\N00:03:11,170 --> 00:03:17,020\NSo you've got all of the main content, the notebook. And then down at the end, you have a big heading appendix.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,33\N00:03:17,020 --> 00:03:20,770\NAnd there you have extra things. You want to make sure you can still run from top to bottom.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,34\N00:03:20,770 --> 00:03:27,490\NBut there you have some of the other things that maybe dove into more details about the building blocks of some of your computations.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,35\N00:03:27,490 --> 00:03:32,650\NBut it is good to show the results after loading data and after doing a complex manipulation,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,36\N00:03:32,650 --> 00:03:38,910\Nespecially one that significantly changes the shape of the data that you're working with.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,37\N00:03:38,910 --> 00:03:41,790\NAnd they talk mostly in this video, though, about markdown sales,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,38\N00:03:41,790 --> 00:03:48,180\Nbecause markdown sells or what you use to build up the structure of your document and make it tell a story,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,39\N00:03:48,180 --> 00:03:52,830\Nnot just be a kind of strange way to present Python code.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,40\N00:03:52,830 --> 00:04:01,710\NSo markdown is a text syntax for simple markup. I'm going to provide a link to the markdown documentation in the class notes that go with this video.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,41\N00:04:01,710 --> 00:04:06,570\NBut there's several inline formatting things. If you put two stars around some text, that'll make it bold.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,42\N00:04:06,570 --> 00:04:16,110\NOne star will make it italics. You can indicate a code using the fit, something that's going to show up as the fixed width code layout using back Tex.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,43\N00:04:16,110 --> 00:04:24,570\NThis is one that I see ignored very frequently in and writing up because it's really,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,44\N00:04:24,570 --> 00:04:27,960\Nreally useful for function names, variable names, things like that.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,45\N00:04:27,960 --> 00:04:32,670\NTo be able to set apart like this is a special thing. This is a function name also.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,46\N00:04:32,670 --> 00:04:39,180\NThen you can use tech math syntax by putting it between dollar signs in this markdown notebook.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,47\N00:04:39,180 --> 00:04:48,450\NPay attention to the details of what your markdown code or what your text formatted text looks like after you render it in the notebook.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,48\N00:04:48,450 --> 00:04:55,500\NMake sure it reads well. Make sure it's clear. Ask yourself if I weren't the one who right wrote this.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,49\N00:04:55,500 --> 00:05:03,290\NWhat? I like reading this and clean it up and pay attention to those details, to make it look,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,50\N00:05:03,290 --> 00:05:08,670\Nto make it look good and to make it be effective at communicating and so that the reader
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,51\N00:05:08,670 --> 00:05:13,170\Ncan clearly understand what the different pieces are and what needs to be emphasized,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,52\N00:05:13,170 --> 00:05:17,260\Netc. Markdown also has a number of block elements.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,53\N00:05:17,260 --> 00:05:21,520\NThe basic one is a paragraph, paragraphs or just text separated by blank lines.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,54\N00:05:21,520 --> 00:05:26,890\NYou can also have bulleted and numbered lists. You can have code blocks for if you need to have a little.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,55\N00:05:26,890 --> 00:05:31,720\NThese aren't super common in a notebook because a lot of your code is in the code cells that you execute.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,56\N00:05:31,720 --> 00:05:34,900\NBut if you need to have a little code that you don't execute for some reason,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,57\N00:05:34,900 --> 00:05:42,970\Nyou can put it in the code block and markdown and then you can also block mathematics, a line on its own that begins and ends with two dollar signs.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,58\N00:05:42,970 --> 00:05:49,170\NAnd you can actually span multiple lines so long as there aren't any blanks that's going to be treated as a piece of block mathematics.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,59\N00:05:49,170 --> 00:05:54,510\NIt's not in line in a sentence, but it becomes its own block and the rendered self.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,60\N00:05:54,510 --> 00:06:00,450\NHeadings are an important one to pay attention to. So Mark Down headings are lines that start with one, two,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,61\N00:06:00,450 --> 00:06:06,510\Nthree up to six hash marks and then a space in the heading text having one heading to hitting three.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,62\N00:06:06,510 --> 00:06:11,370\NSomething that's important to know is the hashes do not mean big and bold.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,63\N00:06:11,370 --> 00:06:16,890\NThat's what they look like. But that's not what they mean. What they mean is heading.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,64\N00:06:16,890 --> 00:06:23,300\NAnd so you need to have an outline structure to your notebook using the headings.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,65\N00:06:23,300 --> 00:06:29,030\NAnd you need to nest them properly, so within each one, you have your H 2s.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,66\N00:06:29,030 --> 00:06:32,760\NAnd then you have your H threes. You don't go straight from H one to H for you.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,67\N00:06:32,760 --> 00:06:41,540\NYou have H three in the middle. Start the notebook with an H one that has the notebook title and that that will become in a lot of rendering context.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,68\N00:06:41,540 --> 00:06:46,850\NThat becomes the title at the top of your notebook. And then all your other headings are two or lower.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,69\N00:06:46,850 --> 00:06:55,670\NAlso you might if you have an appendix, you might have Appendix B, another H1, but also the section headers should be short, not sentences.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,70\N00:06:55,670 --> 00:07:00,630\NIf you're writing an entire sentence in your section header. You're you're putting too much there.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,71\N00:07:00,630 --> 00:07:06,840\NThe section header should be a short title and then the section content comes after it.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,72\N00:07:06,840 --> 00:07:13,650\NNow, one of the few reasons why it's important to use the section headers heading levels properly.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,73\N00:07:13,650 --> 00:07:19,710\NOne is just visually, it helps break up your notebook so we can easily see which component we're at.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,74\N00:07:19,710 --> 00:07:26,130\NSecond, there are extensions that will do things like no your headings or give you a Browsr Bowl table of contents.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,75\N00:07:26,130 --> 00:07:32,130\NYou can use to navigate the notebook by heading what? I'm rendering notebooks as a part of the course website.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,76\N00:07:32,130 --> 00:07:36,960\NYou'll see this over in the right hand side. You can jump directly to notebook headings.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,77\N00:07:36,960 --> 00:07:45,720\NThat only works because I'm consistently using the heading levels to build the structure and outline based structure of my notebook.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,78\N00:07:45,720 --> 00:07:53,160\NAnother a third reason is for accessibility. If someone's reading your notebook with an assistive technology such as a screen reader,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,79\N00:07:53,160 --> 00:08:01,500\Nthe section headings are very important to help them navigate to the parts. The notebook that are both relevant to them at a given time.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,80\N00:08:01,500 --> 00:08:09,180\NSo on the section headers, one additional little rule is if your section editor has to wrap onto a second line, really rethink.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,81\N00:08:09,180 --> 00:08:13,230\NIt's almost certainly too long particularly.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,82\N00:08:13,230 --> 00:08:16,860\NDon't put an entire question in the section. Maybe. Usually.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,83\N00:08:16,860 --> 00:08:22,890\NOccasionally it's OK to put a whole question, but maybe put a brief like three to five word summary of the questions topic and
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,84\N00:08:22,890 --> 00:08:30,750\Nthen write this question itself as the first paragraph of the of the section.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,85\N00:08:30,750 --> 00:08:33,240\NBut pay attention to these different formatting features.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,86\N00:08:33,240 --> 00:08:41,460\NYou can build a well-structured notebook that communicates clearly and draws the reader's emphasis to the places where it needs to go.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,87\N00:08:41,460 --> 00:08:46,650\NWriting the text itself. Use the document to tell a story. What's the goal of what you're doing?
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,88\N00:08:46,650 --> 00:08:51,630\NEither the whole notebook or of individual pieces of analysis. What's the data that we're doing?
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,89\N00:08:51,630 --> 00:08:58,140\NWhat do we know about it going in at the up at the top, either at the very top of your notebook or where you're loading the data?
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,90\N00:08:58,140 --> 00:09:04,050\NIt's useful to write some, especially at the notebooks and we report you submit to somebody.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,91\N00:09:04,050 --> 00:09:07,800\NIt's useful to write there. What do you know? Where did you get the data? How was it collected?
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,92\N00:09:07,800 --> 00:09:16,290\NNot a full data sheet, but at least some summary information to help the reader understand what it is that we're going to be going and looking at.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,93\N00:09:16,290 --> 00:09:19,800\NWhy are we doing each piece of the analysis? What's the purpose here?
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,94\N00:09:19,800 --> 00:09:25,260\NHow does it fit into our broader picture, into our broader goals? What approach are we using?
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,95\N00:09:25,260 --> 00:09:28,200\NWe don't want to just repeat the code writing a a numbered list here.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,96\N00:09:28,200 --> 00:09:33,240\NThe steps and those steps are just a literal translation of the code doesn't help understanding.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,97\N00:09:33,240 --> 00:09:37,570\NIt creates an opportunity for a code and documentation to become mismatched.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,98\N00:09:37,570 --> 00:09:41,670\NBut explain if there's anything tricky in the code.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,99\N00:09:41,670 --> 00:09:48,600\NExplain why that does the job. Explain the conceptual idea behind why you're approaching things the way you are.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,100\N00:09:48,600 --> 00:09:56,700\NIf you're doing a data clean up, explain why that what that cleanup's doing and why that's the right cleanup for your data.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,101\N00:09:56,700 --> 00:10:03,600\NAnd then what do we learn from it? So oftentimes what I do with us, with an individual piece of it, like a chart.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,102\N00:10:03,600 --> 00:10:07,260\NAll right. What question the charge is supposed to be answering.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,103\N00:10:07,260 --> 00:10:11,010\NOr at least the purpose of the chart that we have the code to generate the chart itself.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,104\N00:10:11,010 --> 00:10:18,450\NAnd then we have a tech cell that has observations about what we learn from the chart.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,105\N00:10:18,450 --> 00:10:23,160\NSo what are we doing? How are we going to do it if that's not immediately clear code results?
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,106\N00:10:23,160 --> 00:10:25,590\NAnd then what do we observe from these results?
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,107\N00:10:25,590 --> 00:10:33,350\NSo the over then the high level document structure that I recommend is to start start with the title and intros.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,108\N00:10:33,350 --> 00:10:37,140\NYou've got your title. You're heading one. Then what's the notebook for?
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,109\N00:10:37,140 --> 00:10:41,160\NWhy does this notebook exist? Are there to include links?
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,110\N00:10:41,160 --> 00:10:46,650\NThere's hyperlinks and taxes and markdown as well. Read the markdown documentation to see how to use that.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,111\N00:10:46,650 --> 00:10:50,790\NBut where does this go? Are there things we need to know?
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,112\N00:10:50,790 --> 00:10:54,990\NBackground about where why this documents being created?
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,113\N00:10:54,990 --> 00:11:00,360\NWhere did the data come from? If we have defined research questions, what are those research questions?
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,114\N00:11:00,360 --> 00:11:03,750\NYou can write those right in the intro, the notebook. Then I have a set up.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,115\N00:11:03,750 --> 00:11:09,060\NI almost always have a setup section that comes next. That has input. I import my python libraries.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,116\N00:11:09,060 --> 00:11:11,970\NI've maybe defined some help or functions that I'm going to be using throughout the
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,117\N00:11:11,970 --> 00:11:16,860\Nnotebook helper function specific to one section I might define in that section.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,118\N00:11:16,860 --> 00:11:20,730\NBut then and then I LOEs load the data. Sometimes I load the data as a part of the setup.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,119\N00:11:20,730 --> 00:11:27,990\NSo it's OK. Important modules and then load my data. Sometimes if specially if I have more to say about the data, it's its own section.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,120\N00:11:27,990 --> 00:11:34,500\NBut then as I load each table, I just show the first few rows of it often so that I can see, OK, I've loaded this data and then it's right there.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,121\N00:11:34,500 --> 00:11:39,390\NWe can see as we're going through the rest of the notebook. What is the data just loaded look like?
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,122\N00:11:39,390 --> 00:11:45,030\NThen we perform our analysis and this might be two sections. It might be five, six, seven, eight sections.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,123\N00:11:45,030 --> 00:11:49,920\NAnd then finally at the end, we can summarize and conclude this is going to be really I don't always do this in
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,124\N00:11:49,920 --> 00:11:53,340\Nmy research notebook because often that's the material that goes in the paper.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,125\N00:11:53,340 --> 00:11:57,150\NBut this is going to be something particularly in our assignment, and we're submitting notebooks.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,126\N00:11:57,150 --> 00:12:00,810\NPut that at the end of the notebook. What do we learn from this?
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,127\N00:12:00,810 --> 00:12:03,150\NSometimes they going to have specific directions for things.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,128\N00:12:03,150 --> 00:12:08,730\NI want you to reflect on there when like an assignment one, I've broken down the different requirements.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,129\N00:12:08,730 --> 00:12:15,120\NThose become good candidates for your age to your level. Two headings for each of those.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,130\N00:12:15,120 --> 00:12:21,730\NSo we've got, I think six require six different requirements. An assignment one.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,131\N00:12:21,730 --> 00:12:28,120\NH2, heading a primary section of your document for each of those is a good starting point for your layout.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,132\N00:12:28,120 --> 00:12:34,460\NIn addition to you're probably gonna have another one up at the top for the setup and maybe another for the data load.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,133\N00:12:34,460 --> 00:12:38,590\NBut think about this. This the flow, your document be able to communicate.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,134\N00:12:38,590 --> 00:12:42,760\NWhat are we doing? What are the prerequisites in terms of and data?
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,135\N00:12:42,760 --> 00:12:47,790\NHow are we actually doing it? And then at the end, what do we learn?
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,136\N00:12:47,790 --> 00:12:53,580\NSo you're going to write a lot of cells and produce a lot of outputs in your notebook while you're debugging,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,137\N00:12:53,580 --> 00:12:59,220\Nbefore you submit to before you share in other contexts. Spend some time cleaning up your notebook,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,138\N00:12:59,220 --> 00:13:06,780\Nremove dead ends and extraneous outputs that you included for debugging, but don't fit in the flow of the story.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,139\N00:13:06,780 --> 00:13:13,350\NConsider putting them in a supplementary notebook. If you want to keep them around and then make sure you can rerun your notebook from top to bottom.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,140\N00:13:13,350 --> 00:13:20,300\NSo when the Jupiter interface is the kernel, when you click that and choose, restart and rerun all.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,141\N00:13:20,300 --> 00:13:26,570\NAnd it will restart the python kernel that's actually running your code so all your variables disappear.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,142\N00:13:26,570 --> 00:13:31,220\NYour data is no longer loaded. And then it starts running the notebook from top to bottom.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,143\N00:13:31,220 --> 00:13:37,220\NYou want that to succeed so that someone else working with the notebook can actually rerun and reproduce your results.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,144\N00:13:37,220 --> 00:13:43,790\NIf that doesn't succeed, then that means either you deleted something that's that's important or you're the order of
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,145\N00:13:43,790 --> 00:13:48,740\Nyour source in the notebook does not match the order in which it actually has to be executed.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,146\N00:13:48,740 --> 00:13:54,980\NBut make sure it succeeds and also read back to the notebook to make sure that the charts all still look right.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,147\N00:13:54,980 --> 00:14:01,100\NThe data is the conclusions are all still correct, etc. before you submit the final notebook.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,148\N00:14:01,100 --> 00:14:05,330\NSo when you're writing an up of two, you also need to know your audience and your purpose.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,149\N00:14:05,330 --> 00:14:10,550\NFor example, the notebooks I'm writing for you for teaching purposes here.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,150\N00:14:10,550 --> 00:14:17,060\NThey the things I write in them differ from what I'm going to write in a research notebook that I share with my collaborators,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,151\N00:14:17,060 --> 00:14:22,700\Nor I use my own purposes because my purpose partially in their notebooks, is to explain how they're working.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,152\N00:14:22,700 --> 00:14:28,850\NSo I'm going to say more in these notebooks about how exactly the what exactly the code is
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,153\N00:14:28,850 --> 00:14:35,040\Ndoing is that you can learn how the code works that I would expect in a research notebook.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,154\N00:14:35,040 --> 00:14:41,730\NBut also, you're your own internal your own personal use sharing with your adviser or your supervisor,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,155\N00:14:41,730 --> 00:14:47,970\Nsharing with the public, either the professional public working on your topic or the general public.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,156\N00:14:47,970 --> 00:14:54,180\NThese are all different audiences and they're going to need different levels of explanation and different things highlighted in your notebook.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,157\N00:14:54,180 --> 00:14:56,820\NAlso, not all audiences are well served for notebooks.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,158\N00:14:56,820 --> 00:15:04,500\NNotebooks are fantastic for internal reports, collaboration, et cetera, sharing the results of a data analysis with colleagues or with yourself.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,159\N00:15:04,500 --> 00:15:09,090\NBut for final publication, you're often going to need a separate final report.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,160\N00:15:09,090 --> 00:15:15,930\NI don't know that it's possible to write a research paper and Jupiter notebooks. Somebody might have tried, but.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,161\N00:15:15,930 --> 00:15:21,180\NBut I'll still have the notebook where I explain the analysis. I often make that notebook available.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,162\N00:15:21,180 --> 00:15:31,050\NSo for a lot of my a lot of my published research papers, you can download a zip file or a get repository that contains the notebooks and you
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,163\N00:15:31,050 --> 00:15:36,240\Ncan rerun the experiment and rerun my analysis with the notebooks in the notebook.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,164\N00:15:36,240 --> 00:15:40,370\NThen also I write the files out to disk. And we're not going to see this quite yet.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,165\N00:15:40,370 --> 00:15:45,630\NWe're going to see it later when we start talking about workflow. Because right now I'm just having to submit notebooks.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,166\N00:15:45,630 --> 00:15:49,720\NBut the note, the figures as they show up in the notebook aren't very high resolution.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,167\N00:15:49,720 --> 00:15:55,200\NSo we're gonna want to render a higher resolution version of them to a PMG file or a PDA file or a
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,168\N00:15:55,200 --> 00:16:01,110\Npostscript file that we can then include in our document and word or law tech or whatever we're writing.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,169\N00:16:01,110 --> 00:16:08,910\NSo to wrap up, your notebook is first and foremost a document that contains code to generate the results that you're trying to discuss.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,170\N00:16:08,910 --> 00:16:14,490\NTake advantage of the document structure and use it as a store to tell the story of your analysis.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,171\N00:16:14,490 --> 00:16:19,910\NThe conclusion you come to in why we should believe them. Pay attention to the examples I'm giving you in class.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,172\N00:16:19,910 --> 00:16:36,043\NI'm also going to be trying to give you some examples of research oriented notebooks that you can look at to see examples of good notebook practice.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,