0:00:01.023,0:00:03.411 Hi, can everyone hear me okay? 0:00:03.411,0:00:06.577 Okay, so welcome back to 0:00:06.577,0:00:09.350 the missing semester of your CS education, 0:00:09.350,0:00:12.410 today we're having as a [br]lecture topic "potpourri" 0:00:12.410,0:00:15.170 It's gonna be some miscellaneous[br]combination of topics 0:00:15.170,0:00:18.820 that we the instructors find[br]that are interesting 0:00:19.000,0:00:21.201 But none of them were on their own lecture 0:00:21.201,0:00:24.900 because they're certain topics that [br]we just want you to know about 0:00:24.900,0:00:27.510 because they can be really helpful 0:00:28.639,0:00:33.058 And since we're not gonna delve into[br]a lot of detail in the topics 0:00:33.059,0:00:36.972 If you're more interested about them just[br]feel free to come and ask us questions 0:00:36.972,0:00:40.870 at the end or as we go over [br]them in the lecture 0:00:40.870,0:00:45.660 So the first thing I want to talk about [br]is keyboard re-mappings 0:00:46.180,0:00:48.456 So by now you probably realized that 0:00:48.456,0:00:52.469 we have encouraged you to use the[br]keyboard as your main input method 0:00:52.469,0:00:55.149 so for example, when we went[br]into the editors lecture, 0:00:55.149,0:00:59.639 one of the main ideas of Vim was using[br]your keyboard as much as possible 0:00:59.640,0:01:05.109 so you don't have to rely on going to the[br]mouse because going to the mouse is slow 0:01:05.209,0:01:08.460 And the thing is your keyboard, as with[br]many things in your computer 0:01:08.460,0:01:11.600 is nothing kind of magical,[br]it can be configured 0:01:11.600,0:01:16.739 And it's worth configuring because a lot[br]of the defaults might not be optimal 0:01:16.990,0:01:22.140 The most simple modification that you[br]can do is just to remap keys 0:01:22.450,0:01:25.594 So one of the things we alerted[br]in the editor lecture 0:01:25.594,0:01:28.430 is the Caps Lock key is a really good key 0:01:28.430,0:01:31.049 because it's kind of right[br]in the home row 0:01:31.049,0:01:35.250 and it's kind of large and there,[br]but it's not useful 0:01:35.250,0:01:38.500 You'll probably realize that you don't[br]use your Caps Lock as often 0:01:38.500,0:01:40.200 as when you'd want to use that 0:01:40.200,0:01:42.550 So you can just remap your [br]Caps Lock key 0:01:42.550,0:01:44.450 to something more useful as we mentioned 0:01:44.450,0:01:46.322 like Escape if you're a Vim user 0:01:46.322,0:01:49.050 or like Control if you're an Emacs user 0:01:49.050,0:01:52.180 or useful re-mappings, like a lot of the 0:01:52.180,0:01:56.006 upper row 'F' function keys,[br]or like Print Screen 0:01:56.006,0:01:58.649 You can remap them for example[br]to your media key 0:01:58.649,0:02:01.079 so like when you type Print Screen, 0:02:01.079,0:02:03.613 you probably don't have to [br]do Print Screen that often 0:02:03.613,0:02:06.959 but you probably want to play[br]or pause your music 0:02:07.940,0:02:11.272 And pretty much every operating system has 0:02:11.272,0:02:14.269 some tools that you can use[br]to configure this 0:02:14.269,0:02:16.343 I'm not gonna go into the details 0:02:16.343,0:02:19.460 but there's some of them[br]listed in the notes 0:02:20.080,0:02:23.140 Erm... Let me check 0:02:23.140,0:02:26.597 Oh yeah, another thing that you can [br]do with keyboard re-mappings is 0:02:26.597,0:02:29.160 that you can do more complex combinations 0:02:29.160,0:02:33.360 You can have a combination of keys [br]mapped to some action 0:02:33.360,0:02:36.305 So for example, I have keyboard [br]re-mappings that 0:02:36.305,0:02:40.236 whenever I do Ctrl+Enter, I open[br]a new terminal window 0:02:40.236,0:02:42.692 because that's a thing I do[br]fairly often and 0:02:42.692,0:02:45.600 by default on Mac there is no [br]key binding to do that 0:02:45.820,0:02:49.090 or Ctrl+Shift+Enter will open a [br]new browser window, 0:02:49.090,0:02:51.480 another operation that I do[br]on a daily basis 0:02:51.480,0:02:56.160 So I don't have to grab my mouse[br]and go to Chrome to do that 0:02:56.499,0:02:59.699 You can also do remapping to [br]perform actions 0:02:59.699,0:03:02.550 If you don't want to be typing your[br]password all the... 0:03:02.550,0:03:03.807 Sorry, not your password 0:03:03.807,0:03:06.785 Your Email or your password,[br]or for example, your MIT ID 0:03:06.785,0:03:08.920 like, you may not remember it by heart 0:03:08.920,0:03:12.330 then you can just have a[br]keyboard combination that 0:03:12.330,0:03:16.800 will just perform the action of[br]pasting that text 0:03:19.580,0:03:21.377 Lastly, there are more... 0:03:21.377,0:03:24.727 Right now it looks like you just have[br]to do some function of 0:03:24.727,0:03:28.360 "this is the keys that you press and[br]this is the action that happens" 0:03:28.360,0:03:32.233 But actually there are more complex[br]keyboard combinations, and 0:03:32.233,0:03:36.970 as you go through you'll learn[br]so you can do keyboard sequences 0:03:36.970,0:03:39.110 So for example when we were[br]dealing with tmux, 0:03:39.110,0:03:41.168 in tmux there was this notion of 0:03:41.168,0:03:46.097 first you press Ctrl+A or Ctrl+B, like[br]you press some prefix 0:03:46.097,0:03:49.660 and then some other key and[br]that means something 0:03:49.660,0:03:51.406 A lot of these softwares allow that 0:03:51.406,0:03:55.889 So for example, in my keyboard, since[br]I'm not using Caps Lock at all, but 0:03:55.889,0:04:00.219 every so often I have to use when my [br]undergrad has some software that 0:04:00.219,0:04:03.200 relied on Caps Lock for changing modes 0:04:03.200,0:04:08.764 Then I can press Shift five times[br]in a row quickly, and then 0:04:08.764,0:04:12.469 this software that is in the middle[br]interpreting these commands and 0:04:12.469,0:04:17.049 remapping to some other, will send a[br]single Caps Lock command for that 0:04:17.840,0:04:20.238 Some more examples of that is that 0:04:20.238,0:04:27.770 I mentioned that you can use your[br]Caps Lock key to map to Escape or Control 0:04:27.770,0:04:29.660 but you actually can remap it to both 0:04:29.660,0:04:31.370 So in my computer when 0:04:31.370,0:04:35.100 I just tap the Caps Lock key, that's[br]interpreted as an Escape 0:04:35.100,0:04:40.060 However if I press it and hold it,[br]this software can understand the 0:04:40.060,0:04:42.128 difference between quickly pressing it and 0:04:42.128,0:04:46.155 just holding it for using in [br]combination with some other key 0:04:46.155,0:04:48.789 and then in that case[br]it's mapped to Control 0:04:48.860,0:04:51.833 So a lot of these more[br]advanced configurations are 0:04:51.833,0:04:55.190 supported on a lot of these tools 0:04:55.580,0:04:59.859 As I mentioned we have a [br]short list of good defaults 0:05:00.190,0:05:05.440 for these programs for[br]Windows, Mac OS and Linux 0:05:05.680,0:05:08.360 Any questions on this topic? 0:05:13.120,0:05:16.820 Okay, now I'm going to cover an[br]unrelated topic to keyboard mappings 0:05:16.820,0:05:17.530 [chuckles] 0:05:17.530,0:05:20.912 We're going to see a lot of these[br]unrelated transitions in this lecture 0:05:20.912,0:05:23.600 And it's the concept of "daemons" 0:05:23.600,0:05:25.209 So probably you have... 0:05:25.209,0:05:28.502 Maybe if you are not familiar with [br]the world, it might seem alien, 0:05:28.502,0:05:31.023 but the concept of daemon[br]you're pretty familiar with 0:05:31.023,0:05:33.247 Most computers when you are running them 0:05:33.247,0:05:36.903 there's software that you start and run, 0:05:36.903,0:05:38.905 like the commands that we have been seeing 0:05:38.905,0:05:42.720 You'd type "ls" and then you're[br]calling the List command 0:05:42.720,0:05:46.776 The "ls" command executes because you[br]asked it to execute and then it finishes 0:05:47.296,0:05:51.534 But a lot of other programs are just[br]running as background processes 0:05:51.534,0:05:54.820 and they're just executing[br]in the background 0:05:54.820,0:05:57.385 and waiting for events to happen 0:05:57.385,0:06:00.502 or enabling some sort of functionality[br]in your computer 0:06:01.669,0:06:05.309 Examples of these processes may be[br]like your Network Man[ager], 0:06:05.309,0:06:08.427 like the part of your computer[br]that is managing the network 0:06:08.427,0:06:12.289 or the part of your computer[br]that is managing the display 0:06:12.289,0:06:14.119 Things like that 0:06:14.570,0:06:18.930 You will see that a lot of what [br]is enabled by daemons is 0:06:18.930,0:06:21.410 usually programs that end with a 'd' 0:06:21.410,0:06:25.869 So for instance, when you are [br]SSHing into a computer, 0:06:25.870,0:06:30.399 the receiving computer has to[br]have a SSH daemon 0:06:30.399,0:06:32.639 and the program is called "sshd" 0:06:32.639,0:06:34.860 and if this program is not running then 0:06:34.860,0:06:37.790 there's no way for me[br]to SSH into the computer 0:06:37.790,0:06:41.350 If the program is running, then[br]that program will be listening 0:06:41.350,0:06:44.430 and when you do SSH to that server, 0:06:44.430,0:06:48.202 then some incoming request[br]is gonna enter the computer 0:06:48.202,0:06:52.519 the computer is gonna send it to this[br]daemon that is running in the background 0:06:52.519,0:06:56.280 and then the daemon is gonna check[br]whether you have authorization, 0:06:56.280,0:07:01.113 and if so, it's gonna start some login[br]shell that you can start executing from 0:07:01.403,0:07:06.592 And different OSes handle this[br]somewhat differently 0:07:06.592,0:07:11.409 The main idea is they all have[br]some sort of system daemon 0:07:11.409,0:07:14.816 that responds a lot of[br]these smaller daemons 0:07:14.816,0:07:19.106 In Linux, which is one of the OSes that[br]we are choosing for lot of the examples, 0:07:19.106,0:07:24.290 the tool that you're using[br]is the "systemd" 0:07:24.290,0:07:29.160 Again, first the system daemon is gonna[br]start a lot of these processes 0:07:29.160,0:07:33.380 and if you use this "systemctl" command, 0:07:33.380,0:07:37.460 you can check for the status[br]of different daemons 0:07:37.460,0:07:40.586 you can check for which ones are running, 0:07:43.380,0:07:47.160 you can ask it to start processes,[br]to stop them 0:07:47.160,0:07:49.680 This is kind of an once-off operation 0:07:49.680,0:07:53.387 You can also enable it[br]and then disable them, 0:07:53.387,0:07:57.312 which will tell the system[br]to run them at boot 0:07:57.312,0:08:00.380 or stop running the boot[br]if they were enabled 0:08:01.340,0:08:07.465 And perhaps more interestingly, you can[br]configure your own "systemd" units 0:08:07.465,0:08:13.190 So, so far all the examples are a lot[br]of what the computer has to do, 0:08:13.190,0:08:17.890 but say you want to run a web server.[br]One solution, you could just like, 0:08:17.890,0:08:23.030 every time you start your computer[br]you could open a tmux session 0:08:23.030,0:08:24.880 and then execute the command, 0:08:24.880,0:08:30.050 but that's not really the way that your [br]computer expects daemons to be run 0:08:30.050,0:08:32.666 The way your computer expects[br]daemons to be run is 0:08:32.666,0:08:35.446 by using some sort of "systemd" unit 0:08:35.446,0:08:40.533 It's like a configuration that tells[br]"systemd" how to execute this process 0:08:40.533,0:08:43.960 So an example of this is... 0:08:45.428,0:08:47.912 Here's a very simple example 0:08:47.912,0:08:51.109 So what is happening here is 0:08:51.109,0:08:56.694 we're describing to "systemd" what needs[br]to be done for this program to execute. 0:08:56.694,0:09:00.911 This example is just running[br]a simple Python app 0:09:00.911,0:09:04.854 You can think of it as a web server that[br]can be implemented using 0:09:04.854,0:09:08.349 some Python web server library.[br]And here we're saying 0:09:08.349,0:09:12.546 this is the description, we're[br]saying after, like this is important 0:09:12.546,0:09:16.800 "Systemd" has a list of[br]services that have to start 0:09:16.800,0:09:19.019 Like, all these daemons have to be started 0:09:19.019,0:09:22.815 but maybe there are dependencies [br]between these daemons, so here 0:09:22.815,0:09:27.923 we're saying "no, you should only start[br]this after the network has been set up" 0:09:27.923,0:09:30.669 because otherwise how will you even try to 0:09:30.669,0:09:35.409 configure our web server if I cannot[br]listen to a network port? 0:09:35.409,0:09:38.700 And then we are defining what[br]users should run this, because 0:09:38.700,0:09:42.120 you may want to run this as[br]your user, or maybe other user, 0:09:42.120,0:09:45.360 or maybe the root user[br]should be running this, 0:09:45.360,0:09:48.780 and then what command to run[br]and under what directory 0:09:49.060,0:09:50.761 And whenever you have this, 0:09:50.761,0:09:55.060 there can be small corner cases that[br]you might have to debug, but 0:09:55.060,0:09:58.029 this is kind of the core idea and[br]it can be really useful to 0:09:58.029,0:10:01.339 automate the process of[br]running processes in the background 0:10:01.740,0:10:06.219 A small side note to this is[br]the fact that if you just want 0:10:06.219,0:10:10.936 to run a command every so often, [br]like in some periodicity, 0:10:10.936,0:10:14.639 say every morning I want to[br]do something in my computer, 0:10:14.639,0:10:19.980 you could write a daemon that just does[br]something and then sleeps for a day, but 0:10:19.980,0:10:27.040 actually Linux and Mac OS have already a[br]daemon that does this called "crond" 0:10:27.040,0:10:31.190 and "crond" will take another type of[br]configuration file where you can say 0:10:31.190,0:10:35.660 oh, I want to run a command[br]every day at 8 AM, 0:10:35.660,0:10:38.120 or I want to run a command[br]every 5 minutes, 0:10:38.120,0:10:42.170 and it will just check for[br]this event and execute it 0:10:42.500,0:10:44.640 And with a lot of things[br]you will find that 0:10:44.640,0:10:47.780 there are already daemons that[br]have been configured for that 0:10:48.370,0:10:51.970 Any questions regarding daemons? 0:10:53.920,0:10:57.180 [student talking indistinctly] 0:10:57.760,0:10:59.930 So the question is whether there is 0:10:59.930,0:11:02.700 a folder in the computer[br]where all of these are 0:11:02.700,0:11:04.620 So yeah, like yes and no 0:11:04.620,0:11:09.333 Some of these configuration files are[br]in a couple of different folders 0:11:09.333,0:11:13.260 depending on whether they are[br]system daemons or they are user daemons 0:11:13.260,0:11:18.260 Here you can see at the very first line[br]is where you will place this 0:11:18.260,0:11:22.968 for the system daemon to recognize[br]that it has been installed 0:11:22.968,0:11:25.986 but if you just want to list [br]all the daemons that are running, 0:11:25.986,0:11:30.015 in Linux for example, you can just do [br]"systemctl status" 0:11:30.015,0:11:33.725 and that's gonna print[br]a tree of all the systems 0:11:33.725,0:11:37.882 and which daemon was spawned[br]by which other daemon 0:11:37.882,0:11:42.128 and a lot of them will be[br]spawned directly by "systemd" 0:11:44.150,0:11:49.080 The next topic is going to be[br]file systems in user space 0:11:49.080,0:11:55.798 So, kind of a quick intro to this is the[br]fact that whenever you're using a modern.. 0:11:55.798,0:11:58.260 Oh, yeah, sorry 0:11:58.260,0:12:02.254 And whenever you're using a[br]modern operating system, 0:12:02.254,0:12:07.134 you are not tied to a specific file system 0:12:07.134,0:12:11.480 So modern systems are fairly modular[br]and you can for example, 0:12:11.480,0:12:15.881 in Linux there are different file systems[br]that you can use, and 0:12:15.881,0:12:22.760 the way this works is because the kernel,[br]which is what is running most of the 0:12:22.760,0:12:29.540 operating system, has some modules that[br]know how to interact with a file system 0:12:29.540,0:12:37.493 So usually when you do something[br]like "touch foobar", 0:12:38.890,0:12:44.524 this is happening at a user level 0:12:44.524,0:12:49.655 and then this is going through[br]to the kernel level 0:12:49.655,0:12:54.959 and there is some kind of[br]layer here that is checking 0:12:54.959,0:12:59.940 where this action is happening to [br]figure out what file system it is under 0:12:59.940,0:13:02.628 So for example, you will[br]have multiple disks 0:13:02.628,0:13:05.302 and all the different disks[br]have different file systems, 0:13:05.302,0:13:10.101 so the kernel has to figure out which[br]file system operations to use, 0:13:10.101,0:13:18.499 and say this file might be in an "Ext4",[br]which is the most common Linux one, 0:13:18.499,0:13:23.552 then whenever you do "touch foobar",[br]the kernel will hear that 0:13:23.552,0:13:28.389 and then it will try to figure out like,[br]oh, this lives in an Ext4 file system and 0:13:28.389,0:13:35.939 it will perform the associated instruction[br]for creating a file in an Ext4 file system 0:13:35.939,0:13:39.109 However, the caveat of having[br]a system like this is 0:13:39.109,0:13:44.903 right now I cannot have user code[br]that defines how to create a file 0:13:44.903,0:13:47.919 and that might be kind of[br]useful in some cases 0:13:47.919,0:13:52.736 Say I want to have a file system that[br]every time someone creates a file, 0:13:52.736,0:13:57.559 it sends me an email, so I can know that[br]people are creating these files 0:13:57.559,0:14:01.694 Here I cannot modify[br]the kernel to add this 0:14:01.694,0:14:06.019 So the solution to this is[br]something called "FUSE" 0:14:06.019,0:14:13.520 And FUSE is a way of having[br]file systems in user space 0:14:13.520,0:14:15.299 So what FUSE will do is, 0:14:15.299,0:14:24.859 if this file instead of being in Ext4,[br]if this file is in a FUSE file system, 0:14:24.859,0:14:31.609 FUSE will forward this operation to[br]some other part of user call 0:14:31.609,0:14:35.342 that will say "oh, create this file" 0:14:35.342,0:14:40.617 And here I can have the part of the code[br]that sends an email to me, 0:14:40.617,0:14:43.485 saying "oh, this file has been created" 0:14:43.485,0:14:46.839 And in case you want to[br]still create the file, 0:14:46.839,0:14:53.649 it can forward back the request[br]to do some more kernel operations 0:14:54.199,0:14:58.279 It might not seem really practical,[br]but this is just the theory 0:14:58.279,0:15:01.794 In practice, why this is useful is because 0:15:01.794,0:15:06.240 now you can have user level code[br]that executes arbitrary actions 0:15:06.240,0:15:09.220 when you try to perform[br]file system operations 0:15:09.640,0:15:15.057 A really Interesting example of this[br]is called "SSHFS" 0:15:19.647,0:15:22.526 On an SSHFS FUSE file system, 0:15:22.526,0:15:27.984 whenever you try to create,[br]open, read, write to a file, 0:15:27.984,0:15:30.669 instead of trying to do that[br]to a local file, 0:15:30.669,0:15:35.158 it has an SSH connection[br]to a remote server 0:15:35.158,0:15:39.309 So if I try to create a file here,[br]It will use that SSH connection to 0:15:39.309,0:15:42.793 forward that operation to[br]the remote system 0:15:42.793,0:15:44.796 and then it will perform it there 0:15:44.796,0:15:50.402 So to all my local computer, to the rest[br]of the programs running in my computer, 0:15:50.402,0:15:54.300 there is this path that looks[br]like it is here, but 0:15:54.300,0:15:58.218 all the operations that are performed[br]to the path are forwarded 0:15:58.218,0:16:00.040 to the remote file system 0:16:00.040,0:16:04.222 And with this idea, you will get[br]some examples in the notes, 0:16:04.222,0:16:07.830 and you will find more online,[br]of ways people have 0:16:07.830,0:16:11.065 leveraged this capability to do[br]fairly interesting file systems 0:16:11.065,0:16:13.981 So for example, if instead of having SSH, 0:16:13.981,0:16:18.650 you don't care about SSH because[br]you use Dropbox or Google Drive, 0:16:18.650,0:16:24.210 it's fine, people have implemented[br]FUSE file systems that will mount locally 0:16:24.210,0:16:27.300 and every time you try to do[br]an operation locally, 0:16:27.300,0:16:30.790 actually it goes to one of these[br]cloud storage providers 0:16:30.790,0:16:35.110 so you can also use something like[br]Amazon S3, or Google Cloud Storage, 0:16:35.110,0:16:42.046 that don't have the same[br]kind of UI system 0:16:42.046,0:16:45.400 that we synchronize, as[br]Dropbox or Google Drive 0:16:45.980,0:16:50.906 Another application of this that is not[br]related to doing something remotely, 0:16:50.906,0:16:53.148 is something like[br]an encrypted file system 0:16:53.148,0:16:58.140 You may have a file system that[br]every time you try to write to a file, 0:16:58.140,0:17:02.362 you will try to write it in plain text,[br]but it will capture that operation 0:17:02.362,0:17:04.607 it will encrypt on the go, 0:17:04.607,0:17:07.699 and then it will save it as a[br]regular file in your file system, 0:17:07.699,0:17:09.559 but that's actually encrypted 0:17:09.559,0:17:15.058 And once you dismount the file system,[br]once you remove the FUSE connection, 0:17:15.058,0:17:19.311 all that is left in your computer are just[br]regular files that are encrypted 0:17:19.620,0:17:27.401 The last topic I wanna cover is backups[br]and some good practices about them 0:17:28.630,0:17:34.129 The main idea is that for every[br]file that you care about, 0:17:34.129,0:17:36.300 if you don't have a backup of that file, 0:17:36.300,0:17:38.500 if you don't have a backup[br]stored of that file, 0:17:38.500,0:17:40.470 you can pretty much lose it at any moment 0:17:40.470,0:17:42.803 There are many different failure scenarios 0:17:42.803,0:17:49.170 One of them is just hardware failure.[br]So your hard drive can fail at any moment 0:17:49.170,0:17:54.510 So if you are just making a copy of your[br]files in the same drive, that's not useful 0:17:54.510,0:17:57.850 If your hard drive fails,[br]the files are gone 0:17:57.850,0:18:02.112 The same goes if you have an external[br]drive where you are making a copy, 0:18:02.112,0:18:06.813 but if you are storing everything in your[br]home, and your home burns down... 0:18:07.024,0:18:11.639 Which yes, it's unlikely, but[br]if it happens you just lost all your data 0:18:11.639,0:18:15.779 So you'd have some sort of[br]off-site backup for having this solution 0:18:17.170,0:18:19.679 Another thing to take into account is that 0:18:19.679,0:18:22.949 synchronization or mirroring options[br]are not backups 0:18:22.949,0:18:25.943 So Google Drive, Dropbox[br]that I was mentioning, 0:18:25.943,0:18:30.250 they will just propagate whatever is[br]happening in your computer 0:18:30.250,0:18:33.080 This goes also for[br]hardware mirroring, like RAID 0:18:33.080,0:18:34.765 They are just making a copy 0:18:34.765,0:18:36.840 If you accidentally delete a file, 0:18:36.840,0:18:42.060 or someone maliciously deletes your files,[br]or encrypts them using some ransomware, 0:18:42.060,0:18:46.600 then you might have a copy, but[br]you have a copy of the same useless data 0:18:46.600,0:18:50.857 You actually have to have a solution of[br]how you're running your backups 0:18:50.857,0:18:53.048 And you should be asking yourself 0:18:53.048,0:19:01.280 what actually someone needs to know/have[br]about you in order to delete all your data 0:19:01.280,0:19:09.358 And we have linked different softwares[br]in the notes about how to do this 0:19:09.358,0:19:13.476 The last thing I want to mention about[br]backups is that a lot of the time 0:19:13.476,0:19:16.670 when you think about backups, you just[br]think about the local files 0:19:16.670,0:19:21.070 and like, all my photos and my tax return,[br]and how can I make a backup of that? 0:19:21.070,0:19:25.249 But increasingly in the modern age,[br]there are more and more web applications 0:19:25.249,0:19:28.729 and a lot of data might only live[br]in some cloud provider, 0:19:28.729,0:19:30.570 like for example if you have Webmail, 0:19:30.570,0:19:32.868 and you're not synchronizing it[br]to your computer, 0:19:32.868,0:19:36.740 It's only living in[br]that provider's servers 0:19:36.746,0:19:38.994 And if you don't have a copy for that 0:19:38.994,0:19:41.740 and for some reason you[br]lose access to that account 0:19:41.740,0:19:44.460 because you forgot your password,[br]you got hacked, 0:19:44.460,0:19:49.120 they think you have violated the[br]terms of service... All that data is gone 0:19:49.120,0:19:53.210 So you should look into some tools[br]that people have developed 0:19:53.210,0:19:57.640 for making offline copies of all that data 0:19:57.640,0:20:00.520 so you can make regular backups of that 0:20:00.780,0:20:04.900 And that kind of ends this[br]short section on backups 0:20:04.900,0:20:07.100 Any questions so far? 0:20:09.620,0:20:12.560 (student) When you said that[br]a hard drive can fail at any time 0:20:12.560,0:20:15.460 Is there a reason for it to fail? 0:20:15.460,0:20:16.960 [unintelligible] 0:20:16.960,0:20:21.220 Like if I have my external hard drive[br]sitting at my parents house or something 0:20:21.220,0:20:26.280 And my computer here, is that enough?[br]Or any drive can just fail? 0:20:26.420,0:20:30.370 (Jose) Any drive can fail at any moment.[br]Like we don't... 0:20:30.370,0:20:34.006 Different media have[br]different rates of failure 0:20:34.006,0:20:36.680 and there are really good[br]statistics online 0:20:36.680,0:20:37.755 So for example, 0:20:37.755,0:20:42.370 spinning hard drives have a higher[br]rate of failure than SSDs for example, 0:20:42.370,0:20:43.920 like Solid-State Drives 0:20:43.920,0:20:47.400 And what's another case?[br]Or like CD drives for example 0:20:48.280,0:20:54.360 But if you drop a hard drive it's a[br]higher rate of that failing, of course 0:20:54.360,0:20:59.784 But in general we don't really have[br]an end-all solution 0:20:59.784,0:21:02.140 for saying "this media is not gonna fail" 0:21:02.140,0:21:08.352 Like pretty much, like SD cards, SSDs,[br]hard drives, CDs, degrade with time 0:21:08.360,0:21:12.483 Pretty much every data is kind of[br]bound to this degradation 0:21:12.483,0:21:16.149 or like this fact that it could[br]be lost at any moment 0:21:16.149,0:21:19.156 And you should also know that[br]data can become corrupted, 0:21:19.156,0:21:23.869 your disk might look like it's okay,[br]but maybe some files were corrupted 0:21:23.869,0:21:26.230 and something like[br]synchronization techniques, 0:21:26.230,0:21:30.450 like Google Drive or Dropbox,[br]will propagate that corruption 0:21:30.450,0:21:35.000 And by the time that you realize that[br]things have gone wrong it's maybe too late 0:21:39.580,0:21:46.480 (Jon) Alright, we're gonna continue this trend of jumping between random topics and talk about APIs 0:21:46.660,0:21:52.580 So so far we've really been talking about how do you do things more efficiently locally on your computer? 0:21:52.640,0:21:58.860 Like I want to accomplish this task more efficiently. How do I like configure my editor? How do I use my shell? 0:21:58.900,0:22:03.520 But one thing you should realize is that very often you can integrate with the outside world as well 0:22:03.660,0:22:08.040 Most services that you interact with in your day-to-day provide some kind of API 0:22:08.160,0:22:12.320 For you to interact with the data that they store or the services that they provide 0:22:12.400,0:22:15.400 And usually those APIs are pretty well documented 0:22:15.500,0:22:22.080 If you looked at the APIs for things like Facebook, or Twitter, or Google Drive, or Gmail 0:22:22.140,0:22:28.920 Many of these have interfaces that you can interact with in order to use those services from your local machine 0:22:29.140,0:22:34.200 What's really neat is that you can often combine this with some of the stuff that we've talked about in lecture so far 0:22:34.280,0:22:38.160 Like for example in the Data Wrangling lecture, we looked at how you can create 0:22:38.220,0:22:43.080 These pipelines to extract data from some source that has a different format than you expected 0:22:43.600,0:22:47.720 So for instance, the US government has a free service 0:22:47.840,0:22:51.880 Where you can request the weather forecast for any given location in the US 0:22:52.200,0:22:55.620 And what you do is there is a URL that you request 0:22:55.820,0:23:01.240 And if you set the right parameters in that URL and then just fetch it, what you get back is JSON 0:23:01.240,0:23:05.720 Which is sort of a well-defined data format that you can then parse 0:23:05.760,0:23:10.840 And you can extract things like your 14-day weather forecast 0:23:10.960,0:23:14.940 And maybe you then pipe that into your shell and produce some kind of like 0:23:15.020,0:23:17.940 Handy alias in your terminal that's just gonna print... 0:23:18.020,0:23:23.460 Of some handy reference for the next 14 days of weather in whatever location you were in 0:23:23.600,0:23:27.500 These are things you can pretty easily construct and there's some notes... 0:23:27.580,0:23:31.960 There are some notes in the notes about how you might go about this 0:23:32.140,0:23:37.840 In general when you interact with these APIs, you're going to be using URLs of one form or another 0:23:37.880,0:23:41.200 And the exact format varies from service to service 0:23:41.260,0:23:44.540 But in general the URL is going to contain some set of parameters 0:23:44.600,0:23:49.900 But ultimately you're just gonna issue a web request to them and you're gonna get data back in some format 0:23:50.000,0:23:54.820 One command you should be aware of for interacting with these types of things is one called "curl" 0:23:54.920,0:23:58.560 So curl is a program that you invoke, you give it a URL 0:23:58.620,0:24:01.260 And it just fetches that URL and gives you back the response 0:24:01.260,0:24:03.980 What you do with that response is entirely up to you 0:24:04.080,0:24:08.660 Maybe you pipe it through a program like "gq". Sorry, "jq" 0:24:08.760,0:24:15.000 So jq is a JSON query tool that lets you take in data that's formatted as JSON 0:24:15.040,0:24:19.660 And then write a query over it to extract data that you're interested in 0:24:19.840,0:24:24.640 And this is one of the ways in which you can layer these tools to extract the data that you're interested in 0:24:25.360,0:24:30.360 Some of these services also require that you authenticate in one way or another 0:24:30.460,0:24:33.420 Like for example if you want to interact with a Facebook API 0:24:33.470,0:24:38.860 You need to have some authenticated token that proves who you are as far as Facebook is concerned 0:24:39.040,0:24:43.740 Otherwise, they can't say whether you're allowed to say, create a post as a given user 0:24:43.860,0:24:47.580 Very often these things are gonna use something called "OAuth", although not always 0:24:47.780,0:24:51.700 And you should look at the documentation for every service you care about 0:24:51.740,0:24:55.740 In general though you will get some kind of secret token back from the service 0:24:55.840,0:24:58.620 That you have to include in the requests that you make to them 0:24:58.720,0:25:04.100 Either in the URL or in additional sort of web headers, which you can also send with curl 0:25:04.220,0:25:10.580 Keep in mind though that these tokens are secret. They are another representation of your user 0:25:10.680,0:25:14.120 And anyone who gets their hand on them can basically pretend to be you 0:25:14.180,0:25:17.420 They can do whatever you can do with that token 0:25:17.420,0:25:21.840 So keep this in mind, don't stick them in your "dotfiles" and then push them onto GitHub 0:25:21.940,0:25:26.360 That will land you in trouble. You should think of them as a password 0:25:26.480,0:25:29.380 There are also really neat tools online for integrating services 0:25:29.380,0:25:32.540 So there's a service called "If This Then That" 0:25:32.660,0:25:38.580 Which basically provides integrations with a bunch of different services and lets you chain them together 0:25:38.660,0:25:41.960 And then also access them partially locally if you wish 0:25:42.000,0:25:44.020 This is something that's worth looking into 0:25:44.040,0:25:49.220 If there's a particular service you would like to interact with in a more efficient manner 0:25:49.520,0:25:52.680 Any questions about APIs? 0:25:55.160,0:25:56.450 Alright 0:25:56.450,0:25:59.940 Switching gears entirely, let's talk about command-line arguments 0:26:00.000,0:26:04.080 So command-line tools. There are a lot of them 0:26:04.080,0:26:08.100 And most of them take different arguments because they do different things 0:26:08.200,0:26:13.400 We've talked about looking at man pages for commands and that will tell you how this particular command works 0:26:13.400,0:26:16.800 What kind of flags and options you might give to it 0:26:16.880,0:26:19.420 And what it actually does when you invoke it 0:26:19.580,0:26:23.400 But there are some common themes that are useful to know about 0:26:23.400,0:26:29.400 Either in arguments that many programs take, or just concepts the many of them apply 0:26:29.620,0:26:35.100 The first of these we already mentioned a little bit in the lecture on command-line environments 0:26:35.120,0:26:40.700 Which is the "--help" flag. Very often you can pass this to a program and instead of running, 0:26:40.700,0:26:46.840 It will just print out information about how you can run this program, often in a very short, condensed way 0:26:47.900,0:26:55.040 A similar one is the "--version" flag, which just prints the version of the software you're using 0:26:55.080,0:26:58.000 This can be really handy if you're doing something like filing a bug report, 0:26:58.010,0:27:00.010 Which Anish is going to talk a little bit about later 0:27:00.060,0:27:04.460 And you want to report what version you're running on in case the bug has been fixed since 0:27:04.720,0:27:10.460 Often you can also do "-V" and that means the same as version. But again, check the man page 0:27:11.500,0:27:16.680 There's also "--verbose", or "-v" often 0:27:16.760,0:27:20.760 Which is a flag that lets you increase the output of the program 0:27:21.000,0:27:26.260 It makes the program print more about what it is doing. And very often you can repeat this flag 0:27:26.320,0:27:31.340 So you can do like "-vvvvv" in order to get more information from that tool 0:27:31.460,0:27:34.740 And this can be especially useful if you're trying to debug a problem 0:27:34.760,0:27:39.140 If you're like running "rsync" and you want to know why did it decide to copy this file? 0:27:39.140,0:27:44.120 Or why did it decide not to copy this file? That kind of debug output can be useful 0:27:44.200,0:27:47.960 And often there's a sort of an inverse flag called "quiet" or "silent" 0:27:48.000,0:27:52.880 Which means that the tool will not print anything unless it was like an error 0:27:53.000,0:27:55.140 Anything else, it will stay quiet about 0:27:56.120,0:27:59.420 Many tools, especially those that do destructive actions 0:27:59.480,0:28:03.440 Or some kind of action that you cannot undo 0:28:03.520,0:28:06.320 Provide what's known as a "dry run flag" 0:28:06.600,0:28:09.800 Exactly how this is represented in the command line varies from tool to tool 0:28:09.860,0:28:16.880 But essentially what this dry run mode will do is it will run the tool, but it will not actually make any changes 0:28:17.040,0:28:23.020 Instead it will just inform you of what it would have done if you hadn't run it with dry run 0:28:23.920,0:28:30.400 Many of these tools also have an interactive mode. So for example, th "rm" and "mv" tools both do 0:28:30.540,0:28:33.200 Often just "-i", although not always 0:28:33.320,0:28:35.660 When you run a tool in interactive mode 0:28:35.660,0:28:40.340 It will usually prompt you whenever it's about to do an action that you can't undo 0:28:40.460,0:28:44.380 And it will sort of prompt you for a confirmation that it should actually go ahead 0:28:46.700,0:28:50.100 When we're talking about destructive tools 0:28:50.100,0:28:52.700 Many of them are non-recursive by default 0:28:52.780,0:28:57.000 If you try to remove a directory or you try to operate on a full directory 0:28:57.220,0:29:00.260 They will not continue into the files inside of that directory 0:29:00.480,0:29:05.180 The reason being you might accidentally like, remove your entire hard drive and that seems bad 0:29:05.560,0:29:08.100 Therefore for many of these tools, they have a "recurse" flag 0:29:08.220,0:29:10.680 Often "-r", but again not always 0:29:10.780,0:29:17.180 Which lets them traverse down into the tree to go deeper, but you need to opt-in to this behavior 0:29:17.180,0:29:21.760 So this is for example, the case for "rm". This is also the case for "cp" 0:29:23.660,0:29:28.220 In many tools, when they ask you to give a file name or a path 0:29:28.260,0:29:31.440 And we talked about this a little bit in the data wrangling lecture 0:29:31.440,0:29:34.720 Instead of giving a file name, you can often just give a dash 0:29:34.780,0:29:40.000 Just a single "-", and what that means is standard input or standard output 0:29:40.040,0:29:43.280 Depending on whether that argument is an input file or an output file 0:29:43.380,0:29:48.860 This is handy to know about if you're trying to construct those kinds of data wrangling pipelines that we've talked about before 0:29:49.100,0:29:54.860 Many tools will also default to using standard in or standard out if you don't give any file name at all 0:29:57.400,0:30:03.480 Sometimes you want to pass something that looks like a flag or an option to a command 0:30:03.620,0:30:07.120 But you don't actually want it to be interpreted as a flag or an option 0:30:07.180,0:30:14.700 Consider for example, if you wanted to remove a file called "-i", what would you do? 0:30:14.820,0:30:17.520 Right, if you write the following command... 0:30:19.460,0:30:22.020 "rm -i" 0:30:22.160,0:30:28.860 Well, "-i" is a flag to "rm", so "rm" would, when you run this command say 0:30:28.940,0:30:32.620 Tell me what file to remove, you haven't given me a file 0:30:32.880,0:30:35.580 And it's because it interprets this as a flag 0:30:36.140,0:30:38.500 Similarly, if you do something like... 0:30:38.780,0:30:48.760 "ssh some machine, some command, and let's say, dash r" 0:30:49.740,0:30:56.400 So this is saying - run command "foo" on this machine over SSH, and I wanna pass "-r" to "foo" 0:30:57.180,0:31:02.460 Well, the way that both of these are gonna get interpreted is that these are flags 0:31:02.580,0:31:07.180 Or in this case, this is a flag. But to this command 0:31:07.960,0:31:10.540 Which is probably not what you expected 0:31:10.920,0:31:14.500 Actually in the case of SSH, it has some weird special behavior for some of these 0:31:14.649,0:31:18.089 But often if you want something to not be interpreted as a flag 0:31:18.090,0:31:22.439 There's a very simple way to opt-out of that, and that is using double dash 0:31:22.840,0:31:26.020 If you use double dash, what you tell the command 0:31:26.120,0:31:30.500 Is that verything following this, you should not interpret 0:31:31.000,0:31:36.479 So it will not be considered a flag or an option. In the case of "rm" you can do this 0:31:36.820,0:31:40.480 And now "rm" you will see that the first argument is a "--" 0:31:40.940,0:31:44.520 And then it will keep reading arguments, but it will not interpret them as flags 0:31:44.820,0:31:50.720 So when it gets the "-i", it will not interpret it as the dash i flag, but just as an argument dash i 0:31:51.520,0:31:58.580 Similarly, for SSH you can do this to indicate that these are both positional arguments 0:31:58.700,0:32:04.020 They are not flags or options and you should not interpret things that start with a dash 0:32:08.720,0:32:12.920 (student) But if you do like "--version", it's not gonna trigger that? 0:32:13.180,0:32:17.940 (Jon) No, so this is a " -- ", with a space on both sides 0:32:20.220,0:32:25.040 Any questions about any of this sort of command line conventions business? 0:32:28.840,0:32:30.920 Alright then let's talk about window managers 0:32:31.809,0:32:35.428 So most of you are used to some kind of drag-and-drop window manager 0:32:36.039,0:32:42.599 If you're running Windows, or Mac OS, or Ubuntu - what comes with the machine is like, there are windows and 0:32:42.600,0:32:47.100 They overlap partially on screen and you can like drag-and-drop and move them around and resize them and stuff 0:32:47.820,0:32:51.120 And that works fine 0:32:51.160,0:32:56.200 But it is not the only way to manage windows on your computer it turns out 0:32:56.740,0:33:02.309 So what you are used to is something called a floating window manager, but not all window managers are floating 0:33:03.070,0:33:09.810 Often you can opt-in to other types of window managers that have different behavior for how they arrange your desktop 0:33:10.600,0:33:13.540 A common alternative is a tiling window manager 0:33:13.640,0:33:17.500 So in a tiling window manager, rather than having floating windows, 0:33:17.760,0:33:23.840 Everything is set up into a tiled layout. When you start a program, its window is maximized 0:33:24.180,0:33:28.400 If you start another program, the original window shrinks in size 0:33:28.400,0:33:32.280 And then the new window takes up some subset of the total desktop space 0:33:32.460,0:33:36.660 At no time is your desktop background visible unless you have no programs open 0:33:37.150,0:33:42.150 All of the programs you have open on any given desktop are going to share that space 0:33:42.700,0:33:48.220 This looks a little bit like tmux panes, like we talked about earlier, where you can sort of split them in various directions 0:33:49.140,0:33:54.640 And one of the reasons why this is handy is it means you basically never need to go to your mouse 0:33:54.940,0:33:58.940 In order to move between windows, there are keyboard shortcuts to move to different windows 0:33:58.940,0:34:02.960 There are keyboard shortcuts for resizing the windows or swapping them around on screen 0:34:03.100,0:34:07.380 And this turns out to be a pretty efficient way to manage windows in your computer 0:34:10.149,0:34:15.059 Well, I won't go into too much of detail of what kind of window managers you might use 0:34:15.060,0:34:19.320 Just know that these exist out there and they're worth giving a look. They can be a lot more efficient to work with 0:34:20.460,0:34:23.100 Questions about window managers? 0:34:24.420,0:34:28.120 All right, VPNs. Totally related to the previous topic 0:34:28.440,0:34:32.500 So VPNs are like all the rage these days and this makes me really sad 0:34:32.700,0:34:35.660 It's not clear that VPNs are all the rage for any good reason 0:34:36.180,0:34:39.840 Because you should be aware of what a VPN does and does not get you 0:34:40.480,0:34:47.700 A VPN, in the best case, is really just a way for you to change your internet service provider 0:34:48.040,0:34:54.420 It's a way for you to make traffic on the Internet seem like it's coming from somewhere else than where you actually are 0:34:57.480,0:35:00.780 While that might seem attractive for certain purposes, 0:35:00.780,0:35:04.040 It's a little unclear what it buys you in terms of security 0:35:04.060,0:35:07.480 Because all you're really doing is shifting who you are trusting 0:35:07.600,0:35:11.220 Rather than trusting who is providing your current internet service, 0:35:11.240,0:35:16.340 You're trusting that whatever business is giving you that VPN service... 0:35:16.420,0:35:20.240 You're trusting that they, first of all, have set up this VPN business correctly, 0:35:20.240,0:35:23.820 But also that they are not tracking what you are doing 0:35:24.040,0:35:28.780 And it's not clear whether that change of trust is actually worth it 0:35:28.860,0:35:33.560 If you're sitting at some like dodgy public Wi-Fi network then maybe 0:35:33.640,0:35:36.460 But if you're sitting at MIT, it's not clear 0:35:36.460,0:35:40.700 Do you trust your VPN provider more than you trust MIT's IS&T? 0:35:40.700,0:35:44.100 Or maybe you do, but that is a decision that you need to make 0:35:44.120,0:35:47.400 About what you trust, who you trust and why? 0:35:47.740,0:35:51.680 You should also know that much of your traffic, 0:35:51.720,0:35:57.280 Especially the stuff that's on a sensitive nature on the Internet, is already encrypted 0:35:57.300,0:36:03.520 Whether that's sort of HTTPS or other protocols that use something like TLS, a lot of the sensitive data is already 0:36:03.620,0:36:09.580 Sent over encrypted channels and it doesn't really matter who your network provider is if you're on a dodgy Wi-Fi network 0:36:09.580,0:36:15.220 The stuff that matters is probably encrypted anyway. Might not be, but if it's not 0:36:15.280,0:36:20.580 Then your VPN provider can also see it in plain text just as much as whoever's hosting this dodgy Wi-Fi network 0:36:21.120,0:36:25.780 And notice that I said "in the best case above" 0:36:25.780,0:36:31.820 There are VPN providers who have been shown to be malicious, that do logging of all your traffic, 0:36:31.880,0:36:38.380 That sell that traffic to third parties. There are VPN providers that have forgotten to enable encryption on the VPN 0:36:38.480,0:36:41.640 All of these are real problems 0:36:41.800,0:36:48.800 And so you should think very carefully about whether a VPN actually serves any good purpose for you 0:36:48.980,0:36:52.620 Questions about VPNs? Yes? 0:36:52.740,0:36:56.100 (student) So I have a question about public Wi-Fi networks, because 0:36:56.140,0:37:02.800 The traffic from your computer to the router isn't encrypted between the computer and the router, right? 0:37:02.920,0:37:06.880 Except for what normally is via HTTPS and [unintelligible] 0:37:06.980,0:37:13.640 So then doesn't that mean that people could sniff out what domains I'm going to via the DNS request? 0:37:13.700,0:37:15.700 (Jon) So, it's a very good question 0:37:15.720,0:37:22.080 If you're on a public Wi-Fi network, then the traffic between you and the wireless access point is not encrypted 0:37:22.280,0:37:27.180 At least it's not encrypted sort of on the outer layer, but it might be encrypted in like HTTPS, for example 0:37:27.360,0:37:34.069 And it is totally true the people observing that Wi-Fi network will be able to see anything that is not encrypted 0:37:34.100,0:37:39.760 But the solution to that is to encrypt all your traffic, rather than necessarily going through a VPN 0:37:39.860,0:37:46.820 So one way to do this for example is to use DNS over TLS or DNS over HTTPS, which gives you 0:37:46.920,0:37:51.540 A way to actually encrypt even information that might otherwise leak in plain text 0:37:51.960,0:37:55.880 Rather than try to sort of trust some provider to do that for you 0:37:55.920,0:38:02.450 Now that said, in some cases you might have a trusted institution that provides a VPN network for you 0:38:02.450,0:38:09.080 So for example, MIT provides a VPN network for all MIT students and staff that you can sign up to use 0:38:09.120,0:38:13.780 And in that case you probably trust MIT more than the other networks you might be on 0:38:13.780,0:38:18.020 So it might be worth it, but it's something for you to think about 0:38:18.020,0:38:22.500 (student) When you say you could encrypt it with, what was it, DNS, how would you do that? 0:38:24.180,0:38:28.700 (Jon) So, DNS is the way that people turn domain names, 0:38:28.700,0:38:32.300 Or your computer turns domain names into IP addresses to know what computer to connect to 0:38:32.460,0:38:37.180 And that protocol by default is in plain text. There's nothing encrypted about it 0:38:37.200,0:38:40.940 There are various ways to encrypt your DNS traffic 0:38:41.060,0:38:43.940 Some of them are standardized and some of them are not 0:38:44.000,0:38:49.200 I won't go into the exact mechanics here, but you should google it and look at some of the ways 0:38:50.100,0:38:52.680 Okay. The last thing I want to talk about is Markdown 0:38:52.820,0:38:59.280 So there is a high chance that some of you are going to write text over the remaining part of your life 0:38:59.480,0:39:02.860 And you will want to mark up that text in various simple ways 0:39:03.030,0:39:08.929 And one thing you could do is start up Word or use LaTeX or something like that to mark up your documents 0:39:08.930,0:39:10.860 But that is a pretty heavy-handed approach 0:39:10.860,0:39:13.320 Instead it would be nice if we could just sort of 0:39:13.400,0:39:16.380 Write things the way we feel like they should be 0:39:16.400,0:39:20.960 I don't know how to describe it in a better way but sort of the natural way where if you want something, 0:39:20.980,0:39:25.280 If you want to put emphasis on a word you just put like stars around it or something, and then it just works 0:39:25.380,0:39:31.760 Markdown is essentially that. It is a way to try to encode the way that we often write text somewhat naturally 0:39:31.860,0:39:37.440 Into a markup language that lets you write things like bold text, links, lists, that sort of stuff 0:39:37.560,0:39:41.540 In fact, all of the lecture notes for this class have been written using Markdown 0:39:41.720,0:39:46.260 And Markdown is really very straightforward. The basic rules are in the notes 0:39:46.300,0:39:53.660 But the basic things you need to know is, in Markdown if you put stars 0:39:53.820,0:40:00.480 Around a word, that word is emphasized. Or some sequence of words. If you put double stars 0:40:00.640,0:40:04.760 That word is emphasized strongly, also known as bold 0:40:04.760,0:40:11.080 There are various other things you could do, like if you put a dash before a line it is now a list 0:40:11.080,0:40:14.180 And there's one list item and you can amend list items 0:40:14.240,0:40:18.540 If you put "1." in front, or some other number, it becomes a numbered list 0:40:18.720,0:40:25.000 If you put a pound sign in front of something it becomes a header, like some kind of title header 0:40:25.120,0:40:30.260 If you put multiple of them, they become subheadings and you can keep adding more to these 0:40:30.640,0:40:37.820 If you want to write code you can put a single backtick, followed by some code, followed by a backtick 0:40:37.880,0:40:41.400 And now that is rendered in Monospaced Font 0:40:41.440,0:40:46.920 If you want multiple lines of code, you do a triple backtick, and then code, 0:40:46.920,0:40:50.960 And then maybe some more code and then triple backtick 0:40:51.240,0:40:53.840 And in many cases like if you're on GitHub for example, 0:40:53.860,0:40:58.440 You can even type the name of a language up here after the backticks, without a space 0:40:58.480,0:41:01.520 And it will be syntax highlighted in the language of your choice 0:41:01.660,0:41:07.900 This is a really handy thing that is supported in so many websites nowadays you might not even realize 0:41:07.900,0:41:11.280 Like in Facebook Messenger you can use many of these 0:41:11.520,0:41:14.400 They don't actually officially say they support Markdown anywhere 0:41:14.440,0:41:17.700 But many of these things just like sort of happen to work and it's worth learning 0:41:17.780,0:41:22.980 At least the basics and just start using them. You can do links and stuff as well, but that's already in the notes 0:41:23.860,0:41:26.400 Any questions about Markdown? 0:41:27.940,0:41:31.800 Right, Anish, you're up 0:41:34.700,0:41:37.040 (Anish) Is my microphone working? 0:41:38.400,0:41:41.160 Is this working? Can you guys hear me in the back? 0:41:42.380,0:41:46.420 The light's green. Oh, I think I can hear it. Okay, great 0:41:48.260,0:41:53.500 So, continuing with our theme of random topics that are all unrelated to the previous topics we've been talking about, 0:41:53.540,0:41:56.840 The next thing we're going to talk about is a program called "Hammerspoon" 0:41:56.850,0:42:00.469 Which is a tool for doing desktop automation on Mac OS 0:42:00.560,0:42:05.160 And I think there's similar tools for Windows and Linux, a lot of the ideas can carry over 0:42:05.200,0:42:08.640 You can google it if you want to figure out how to do these things on other platforms 0:42:08.640,0:42:13.000 But basically Hammerspoon is a program that lets you write Lua scripts, scripts in a programming language 0:42:13.290,0:42:15.830 That interact with various operating system functionality 0:42:15.960,0:42:20.280 So you can write code that interacts with the keyboard and mouse and connects that to window management, 0:42:20.400,0:42:25.980 To display management, the file system, battery and power management, Wi-Fi... All sorts of stuff 0:42:25.980,0:42:30.560 Like basically all the things that your operating system manages, this tool lets you hook into those things 0:42:30.560,0:42:34.310 And so it can let you do all sorts of neat things by writing just a couple lines of code 0:42:34.440,0:42:39.540 Just some examples of cool things you can do with this tool are - you can bind hotkeys to move windows to 0:42:39.680,0:42:45.240 Specific locations. So a demonstration of this is here. I have this window open. I press, in my particular setup, 0:42:45.280,0:42:49.660 "Option+Command+Right" and this window moves to the right. "Option+Command+Left", this window moves to the left 0:42:49.740,0:42:52.360 And I have a couple other shortcuts for moving things to various places 0:42:52.370,0:42:57.499 And so I can kind of have an effect similar to tiling window managers that Jon was talking about earlier 0:42:57.680,0:43:02.870 I can move windows to different parts of my screen to set things up in a particular way rather than have to use the mouse 0:43:02.870,0:43:07.699 To position things where I want them to be and then like click and drag to resize windows to the right shape 0:43:07.800,0:43:10.440 Just a keyboard shortcut can do the trick 0:43:10.460,0:43:15.100 But this tool is not limited to just moving windows around and binding that to particularly keyboard shortcuts 0:43:15.100,0:43:19.000 You can do other things like create a menu bar button with a bunch of different options 0:43:19.040,0:43:22.580 And you can bind those different options to do different things. So in my particular case 0:43:22.640,0:43:28.420 I've created this little menu and then I have a bunch of different things that I do reasonably frequently and clicking on these things 0:43:28.920,0:43:33.440 Invokes a particular Lua function that I've written that interacts with this library. So for example 0:43:34.170,0:43:38.780 Here, this "Rescue windows" thing is a particular thing where I often work with multiple displays 0:43:38.880,0:43:44.660 And sometimes my operating system gets confused and I have some window that ends up off of my display and 0:43:44.740,0:43:47.260 How do I how do I get this thing back? 0:43:47.520,0:43:49.780 Well, that's what this... Whoops, not that 0:43:52.200,0:43:56.600 That's what this "Rescue windows" thing does. It brings windows that are off the screen back onto the screen 0:43:57.620,0:44:01.840 Another neat thing I have setup here is I have particular layouts that I've named 0:44:01.920,0:44:03.640 So like a dorm, and a Lab and a Laptop layout 0:44:03.650,0:44:04.910 So for example in my Lab 0:44:04.910,0:44:08.660 I have this screen and I have another screen and have another screen besides that in a different orientation 0:44:08.760,0:44:11.570 And I have this particular setup that I want where I want 0:44:11.780,0:44:15.110 Maybe my terminal full screen on here, and my chat program over here and 0:44:15.270,0:44:18.409 This screen split up into five segments with different programs in different places 0:44:18.980,0:44:26.460 Here I can, when I show up to Lab I can just go here and click "Layout Lab" and it will invoke some code 0:44:26.760,0:44:32.200 Which is not all that complicated like 10 lines of code describes a particular layout 0:44:32.280,0:44:35.820 And it will instantiate that layout and put all the things where they need to go 0:44:36.100,0:44:42.020 I could even in theory automate some of these things where my computer could figure out like I plug in a display and my computer knows 0:44:42.020,0:44:46.370 Oh, this is the display that you have in your lab. Let me automatically instantiate this layout for you 0:44:46.370,0:44:50.899 That's another thing you can do with Hammerspoon. And also other wacky things you can do like you can do things like 0:44:51.660,0:44:55.040 It can detect your Wi-Fi network that you're on so it knows kind of where you are 0:44:55.040,0:45:00.110 Maybe I have a different Wi-Fi network name at home versus in lab and I can do things like when I show up to lab 0:45:00.420,0:45:04.040 Automatically mute my speakers. So I don't have like embarrassing music play out loud in my lab 0:45:05.280,0:45:07.280 Another kind of cool example is 0:45:07.890,0:45:09.440 So I have a Mac 0:45:09.440,0:45:12.050 It has a fancy power supply and a lot of my friends have 0:45:12.240,0:45:15.469 Computers that look the same as this and their power supply bricks look the same as mine and 0:45:15.720,0:45:18.860 Sometimes I use their power brick because I forgot mine at home or something 0:45:19.110,0:45:21.620 This tool can actually with like three or four lines of code 0:45:21.780,0:45:27.469 Do neat things like show you a warning like it'll pop up a warning if you've accidentally taken your friend's power supply and plugged it 0:45:27.470,0:45:29.470 Into your computer instead of using your own 0:45:29.550,0:45:35.090 So at a high level this tool lets you run arbitrary Lua code and do things like bind it to menu buttons or key presses and 0:45:35.340,0:45:39.169 It interacts with a large part of the operating system in order to do all sorts of cool stuff 0:45:39.600,0:45:42.500 So that is Hammerspoon, any questions about that? 0:45:44.480,0:45:46.520 Cool. Moving on to the next topic 0:45:47.490,0:45:51.199 Completely unrelated to the previous one, it's booting and live USBs 0:45:52.080,0:45:55.279 So the operating system on your computer, Windows or Mac OS 0:45:55.540,0:46:00.790 or whatever you're used to is not exactly the first thing that runs on your machine when it turns on. There's something else that 0:46:00.790,0:46:04.060 happens in the boot process before your operating system is loaded and 0:46:04.670,0:46:08.470 There's some interesting stuff that you can do there. So you might have seen when you turn on your computer 0:46:08.470,0:46:13.330 It says something like press F9 to configure the BIOS or press F12 to enter the boot menu 0:46:13.580,0:46:17.860 The particular key sequences may depend on your machine and specific configuration 0:46:17.860,0:46:22.809 But this is a general pattern and you can configure all sorts of interesting Hardware related stuff here 0:46:22.810,0:46:23.950 So it's worth checking out 0:46:23.950,0:46:29.410 And another thing you can do in this boot menu is you can have your computer start off from an alternate boot device 0:46:29.420,0:46:34.180 so by default like my laptop here has a solid-state drive and it boots Mac OS when it turns on but I can also 0:46:34.250,0:46:40.360 say plug in a USB flash drive that has an operating system installed on the flash drive and then at boot tell my computer to 0:46:40.360,0:46:44.799 Boot from that flash drive instead of the built-in solid-state disk, and this is useful 0:46:44.800,0:46:45.230 for example 0:46:45.230,0:46:47.030 If I've broken my operating system 0:46:47.030,0:46:52.239 Install and I want to do something like get the data off my computer or maybe want to fix the operating system 0:46:52.240,0:46:55.479 Like maybe there's some critical files somewhere that I've deleted or I forgot my password 0:46:55.480,0:46:59.350 I need to go like tweak some files in order to reset it booting from a live USB 0:47:00.020,0:47:04.419 Booting from the separate operating system that's installed on a flash drive can let me do that like boot up my operating system 0:47:04.760,0:47:07.300 Mount the hard disk that's on my current machine 0:47:07.300,0:47:09.939 I'm working on and then go make some tweaks or copy data off of that 0:47:10.340,0:47:16.059 And so live USBs are really useful and in the lecture notes, we've linked to a tool that can help you create them really easily 0:47:16.970,0:47:19.780 Any questions about the boot process or live USBs? 0:47:22.460,0:47:28.760 All right, next topic is virtual machines, Vagrant, Docker, the cloud and OpenStack 0:47:28.760,0:47:31.620 I think last year we had an entire lecture on this topic this year 0:47:31.630,0:47:33.969 We're going to condense it into like one minute 0:47:34.100,0:47:41.229 So at a high level virtual machines and similar tools like containers let you emulate a whole computer system within your current machine 0:47:41.230,0:47:44.439 so like I'm running Mac OS here, but within my Mac OS 0:47:45.380,0:47:46.460 environment I can 0:47:46.460,0:47:49.750 Simulate a machine that's running say Ubuntu or some other operating system 0:47:49.750,0:47:55.179 And this is a nice way of creating an isolated environment for testing or for development or for exploration 0:47:55.700,0:48:00.640 For example doing things like running potentially malicious code that should be isolated from my current environment. I 0:48:01.190,0:48:06.940 Think the most common use case for programmers is to use virtual machines are containers to create development environments 0:48:06.940,0:48:11.919 So I'm using Mac OS and I have some of services and stuff and libraries installed in my current machine 0:48:11.990,0:48:13.280 But I might want for example 0:48:13.280,0:48:18.310 I'm working on some web programming project and I want it to run on an 0:48:18.800,0:48:20.090 Debian machine and I need 0:48:20.090,0:48:24.489 Postgres, like a database server installed rather than install that all on my Mac OS machine 0:48:24.490,0:48:28.869 I can instantiate this new machine just for the development purposes 0:48:30.140,0:48:34.600 Now virtual machines, like that's a general concept. There are a bunch of programs that let you 0:48:35.270,0:48:37.780 that are called virtual machine hypervisors that 0:48:38.330,0:48:44.739 Support this functionality on your machine and then there are tools that let you script these hypervisors in order to specify machine 0:48:45.020,0:48:51.040 Configurations like operating system and like what packages you want installed and what services you want installed in plain text 0:48:51.040,0:48:56.739 And so this is an example on the screen right here. And this is done using a system called Vagrant. 0:48:57.859,0:49:00.789 It's linked in the lecture notes. You can look into this if you're curious. 0:49:00.790,0:49:05.439 So basically and the short plain text file I can specify okay. I want a machine that's running Debian 0:49:05.440,0:49:09.220 It should have "postgres" and Redis and Python and stuff installed on it. 0:49:09.220,0:49:10.880 And then, once I have this configuration, 0:49:10.880,0:49:16.840 I can just type in "vagrant up", and what it does is, it reads this file and instantiates a new machine based on this 0:49:17.000,0:49:22.659 configuration. And then, after I've done that I can do "vagrant ssh" to SSH into this virtual machine. 0:49:22.660,0:49:25.510 So it's not a remote machine running on some other piece of hardware somewhere, 0:49:25.540,0:49:30.790 It's just simulated on my own machine, but now here I have an Ubuntu box like I do now, let's be really stashe 0:49:30.820,0:49:34.629 It's like or sorry not a bunch of debian here with all the things 0:49:34.629,0:49:38.340 I want installed in here and I can do my development inside this isolated environment, 0:49:38.440,0:49:42.240 and not, kind of, install all this junk on my MacOS machine. 0:49:42.980,0:49:50.080 Now so that's vagrant there's similar tools like docker that are conceptually similar but use containers instead of virtual machines 0:49:50.119,0:49:52.719 It's a distinction that we're not going to talk about in too much detail right now 0:49:53.690,0:49:55.690 And so you can run VMs on your own computer 0:49:55.760,0:49:58.570 but you can also rent virtual machines on the cloud and so 0:49:58.730,0:50:02.859 It's a nice way to get instant access to like one example is you might want a computer 0:50:02.859,0:50:07.809 That's always on always connected to the internet and has a public IP address. Like maybe you want to run a web server 0:50:07.810,0:50:12.249 That's always available or you want to run some other service like say a slack bot or something like that 0:50:12.350,0:50:14.000 Well a virtual machine 0:50:14.000,0:50:19.239 rented on the cloud is one nice way to get that and these are pretty cheap for like a low capacity machine with a 0:50:19.430,0:50:21.430 small CP and small amount of disk space and 0:50:22.240,0:50:26.919 You might want to do is get access to a machine that's really powerful like with a lot of CPU cores or with a lot 0:50:26.920,0:50:31.480 Of RAM or with a whole bunch of GPUs for some specific purpose like say you're doing deep learning or so 0:50:31.480,0:50:33.440 You're doing some other sorts of sensitive computation 0:50:33.440,0:50:35.380 Well, that's another thing you can do with VMs on the cloud 0:50:35.380,0:50:41.140 And finally, you can get access to many more machines than you have physical access to. Like if I need a thousand machines 0:50:41.140,0:50:46.540 But only for two minutes to do some very parallel tasks. That's something I can easily do with virtual machines and 0:50:47.840,0:50:51.040 Popular services for doing this are things like Amazon AWS or Google cloud 0:50:51.320,0:50:56.600 And if you're a member of MIT CSAIL, you can also get free VMs for research purposes using the CSAIL OpenStack 0:50:56.720,0:50:59.000 And so this is also linked in the lecture notes 0:50:59.540,0:51:03.459 So any questions about virtual machines, or Vagrant, Docker, or anything like that? 0:51:12.530,0:51:17.229 So the question is when I say I'm running Ubuntu, or, actually, in this case, it's Debian 0:51:18.140,0:51:22.240 when I'm running Ubuntu here, do I have, like, Ubuntu installed on my machine, or 0:51:22.940,0:51:28.060 What exactly is going on here? So, basically, what Vagrant did for me when I type "vagrant up" is, 0:51:28.970,0:51:35.169 because I've specified I want Debian here, it downloaded Debian from the internet, like, set up a disk image for this new machine, 0:51:35.510,0:51:38.919 installed Debian into that disk image, then went to install these programs 0:51:39.080,0:51:44.109 So, like, yes, this is on my computer. But, all of this is just in a particular file that's a disk image. 0:51:44.109,0:51:48.489 And then, I'm emulating a machine that is basically completely isolated from my current machine. 0:51:48.490,0:51:54.909 This is being run as a process on my current machine. Does that answer the question? Any other questions about VMs? 0:51:56.990,0:51:58.100 Great. 0:51:58.100,0:52:00.100 Next topic is also going to be a quick mention - 0:52:00.440,0:52:01.760 So, 0:52:01.760,0:52:05.020 a lot of you are programmers, and you're used to writing programs 0:52:05.180,0:52:07.720 in a tool like Vim, or some other editor that you're comfortable with. 0:52:07.970,0:52:13.000 Another thing that can be really neat to use for particular tasks, is something called a notebook programming environment. 0:52:13.000,0:52:17.469 And this is a more interactive way of writing programs. Here on the screen, I have a demo. 0:52:17.470,0:52:23.439 This is something called Jupiter notebook and it can be used for writing Python programs. I think they also support some other languages. 0:52:24.140,0:52:26.350 And, basically, this is a nice way of doing interactive programming. 0:52:26.390,0:52:29.920 So, normally, you're used to writing a big program in a file, or a collection of files, 0:52:29.920,0:52:35.020 and once you're done writing it, you can just run the whole program. But, this lets you be a little bit more flexible and 0:52:35.619,0:52:39.818 run little snippets of code at a time. Like, for example, I can break my program into these little pieces. 0:52:39.890,0:52:45.969 It's just some random code I wrote. And, I can say, "execute this cell", and I press a particular key combination to execute the cell. 0:52:46.069,0:52:50.798 But then, I can go back, and tweak my program a little bit. Like, say I want to have this be lowercase instead. 0:52:50.799,0:52:54.609 Then, I can execute this cell, and then go and evaluate this thing, and 0:52:55.009,0:52:59.949 this way I can, kind of, run little snippets of code, within a Python environment. 0:53:00.799,0:53:05.649 And, it's a nice way of building up programs, piece by piece, rather than having to write everything at once. 0:53:05.650,0:53:10.749 This is really useful for particular research purposes, like I think a lot of people use these for doing machine learning work, for example. 0:53:13.249,0:53:17.829 Any questions about the idea of notebook programming environments? They're worth checking out. 0:53:24.349,0:53:27.998 Oh, so the question is, "This looks like it's online, is there an offline version of Jupiter notebooks?" 0:53:27.999,0:53:32.679 So, actually, this is the thing that runs in the browser, but it's running locally... 0:53:32.680,0:53:35.499 So, I don't know if you can see it on the screen, because it's kind of small, but up here 0:53:35.499,0:53:37.499 it says, "localhost:8888". 0:53:37.819,0:53:44.349 Here, I have running on my own local machine, a Jupiter notebook. And, they've just built it 0:53:44.349,0:53:46.349 so it runs within the web browser. 0:53:47.029,0:53:49.029 That being said, there are also online 0:53:49.400,0:53:54.729 Jupiter notebooks that you can use, where the Python kernel is actually running on some remote machine. You might want to do this 0:53:54.729,0:53:55.960 for example, like on my laptop, 0:53:55.960,0:54:00.159 I don't have a fancy GPU, but in my room, I have a machine with a fancy GPU. And so, when I'm doing machine learning 0:54:00.160,0:54:07.119 work, I often SSH into that machine, run a Jupiter notebook on there, and then open up the interface, in my local web browser, 0:54:07.119,0:54:09.848 so I have access to that powerful GPU running on my different machine. 0:54:11.900,0:54:13.900 Any other questions? 0:54:16.279,0:54:17.690 Great. 0:54:17.690,0:54:19.690 The final thing we're going to talk about today is 0:54:20.299,0:54:24.488 Github. So, we touched on this a little bit during the version control lecture. 0:54:24.489,0:54:28.119 But, Github is one of the most popular platforms for open-source software development. 0:54:28.400,0:54:35.440 It hosts source code. It hosts git repositories. But, they also have other tools for managing a project. 0:54:36.319,0:54:42.129 And, like, a lot of the tools we've talked about in this class are hosted on Github. For example, 0:54:42.130,0:54:45.609 like, Hammerspoon, the thing we just talked about, is developed on Github 0:54:46.400,0:54:47.930 It's really easy to get started 0:54:47.930,0:54:53.950 contributing to open-source projects on Github to help improve the tools that you use every day. 0:54:54.260,0:54:57.520 There are two primary ways you can contribute to projects on Github. 0:54:59.330,0:55:06.309 Let's open up some repository. We can actually go to the Github repository for the class website. 0:55:06.310,0:55:08.310 So, this is an open-source software project. 0:55:09.050,0:55:10.610 And... 0:55:10.610,0:55:14.799 Let's zoom in a little bit. So the two ways you can contribute to projects on Github - 0:55:14.800,0:55:17.410 The two main ways are through issues and pull requests 0:55:17.480,0:55:23.949 One thing that's actually really helpful to developers, and also pretty lightweight and easy for users to do, is to report issues with a 0:55:23.950,0:55:27.310 software project. Like, say you're using somebody's program and you encounter some bug... 0:55:27.560,0:55:31.989 Writing a high quality issue is actually super helpful to developers and hopefully doesn't take you too much time. 0:55:31.990,0:55:37.510 And so, you can go to here, like find the project on GitHub, go to the issues page, and click on new issue, and then 0:55:37.510,0:55:41.979 write some high quality bug report. And then, hopefully the developer will respond and fix the issue for you. 0:55:42.160,0:55:43.370 So, for example, for this class, 0:55:43.370,0:55:49.120 like, one of the students in this class pointed out an issue with our lecture notes, and after she pointed it out, 0:55:49.120,0:55:54.819 I said, okay, like, that looks like a reasonable thing. Let's fix it. And in this particular case, instead of fixing it myself, 0:55:54.820,0:55:59.620 I actually asked this person, "Do they just want to fix it for me?" And so, that leads into the other thing 0:55:59.620,0:56:06.009 I want to talk about: issues and pull requests. So, pull requests are the second way to contribute to projects on Github. And, 0:56:06.560,0:56:09.880 this involves actually contributing code back to the project. And 0:56:10.550,0:56:16.779 so if we look at the pull request for this particular project, you'll see that a bunch of people have submitted code changes. 0:56:17.510,0:56:18.800 And, 0:56:18.800,0:56:24.160 the process for doing so - so this is showing the difference, the patch, that this person submitted. 0:56:25.190,0:56:30.310 Basically, the process for creating pull requests is a little bit more involved than submitting issues. Like you're not just submitting text: 0:56:30.310,0:56:32.080 you're actually going to modify their source code. 0:56:32.080,0:56:36.489 And so, we've linked to some guides that explain the process in a little bit more detail. 0:56:36.490,0:56:41.860 But, at a high level, what you do is, you take the repository on Github, "fork" it, and then download it locally, 0:56:41.860,0:56:43.040 so now have your own, local copy. 0:56:43.040,0:56:48.130 Then, you can go and work on it, do some development work, and fix a bug or add a feature, and then, eventually, you 0:56:48.130,0:56:51.969 send what's called a "pull request" back to the original developer. So, you say, 'Here. I've made some changes. 0:56:51.970,0:56:54.640 Can you please incorporate them back into the original project?' 0:56:54.640,0:57:00.429 And, after that point, what usually happens with these projects, is that the maintainer will go back and forth with you, giving you feedback on 0:57:00.430,0:57:06.110 the changes you've proposed. And, eventually, once everybody's happy, they will "merge" in your changes, and they'll be available to everybody who uses the project. 0:57:07.500,0:57:14.179 So, that is how you can contribute to projects on GitHub and make software better for everybody. And, so any questions about GitHub? 0:57:19.230,0:57:24.350 Cool, okay. So, that is it for the topics for today. Any questions about the lecture overall? 0:57:25.920,0:57:27.840 Great, okay, so before - 0:57:27.840,0:57:33.350 before we finish, a quick description about tomorrow's lecture: so, today was all the topics we thought are interesting, 0:57:33.350,0:57:34.140 we should talk about, 0:57:34.140,0:57:38.900 tomorrow's lecture is gonna be about all the topics you think are interesting and that we should talk about. So, tomorrow is going to be 0:57:38.900,0:57:40.940 a Q&A lecture. And, after today, 0:57:41.430,0:57:43.170 uh, after the lecture, we'll submit, 0:57:43.170,0:57:48.020 we'll email out a link where you can submit questions for us to answer. And so, please go and fill that out, 0:57:48.020,0:57:50.020 otherwise, we won't have too much to talk about tomorrow. 0:57:51.180,0:57:53.899 Great, so hopefully, see you tomorrow in our Q&A lecture.