WEBVTT 00:00:00.400 --> 00:00:02.860 Okay, welcome back. 00:00:02.860 --> 00:00:05.920 Today we're gonna cover a couple separate 00:00:05.920 --> 00:00:07.620 two main topics related to the shell. 00:00:07.620 --> 00:00:11.240 First, we're gonna do some kind of shell scripting, mainly related to bash, 00:00:11.240 --> 00:00:14.160 which is the shell that most of you will start 00:00:14.160 --> 00:00:18.520 in Mac, or like in most Linux systems, that's the default shell. 00:00:18.520 --> 00:00:22.720 And it's also kind of backward compatible through other shells like zsh, it's pretty nice. 00:00:22.740 --> 00:00:25.940 And then we're gonna cover some other shell tools that are really convenient, 00:00:26.060 --> 00:00:29.320 so you avoid doing really repetitive tasks, 00:00:29.320 --> 00:00:31.580 like looking for some piece of code 00:00:31.580 --> 00:00:33.420 or for some elusive file. 00:00:33.420 --> 00:00:36.160 And there are already really nice built-in commands 00:00:36.160 --> 00:00:40.960 that will really help you to do those things. 00:00:40.960 --> 00:00:43.260 So yesterday we already kind of introduced 00:00:43.260 --> 00:00:46.160 you to the shell and some of it's quirks, 00:00:46.160 --> 00:00:48.720 and like how you start executing commands, 00:00:48.720 --> 00:00:50.600 redirecting them. 00:00:50.600 --> 00:00:52.400 Today, we're going to kind of cover more about 00:00:52.460 --> 00:00:56.120 the syntax of the variables, the control flow, 00:00:56.120 --> 00:00:57.720 functions of the shell. 00:00:57.720 --> 00:01:02.700 So for example, once you drop into a shell, say you want to 00:01:02.760 --> 00:01:06.360 define a variable, which is one of the first things you 00:01:06.360 --> 00:01:09.340 learn to do in a programming language. 00:01:09.340 --> 00:01:12.740 Here you could do something like foo equals bar. 00:01:12.860 --> 00:01:18.400 And now we can access the value of foo by doing "$foo". 00:01:18.460 --> 00:01:21.400 And that's bar, perfect. 00:01:21.400 --> 00:01:24.480 One quirk that you need to be aware of is that 00:01:24.480 --> 00:01:27.900 spaces are really critical when you're dealing with bash. 00:01:27.900 --> 00:01:33.380 Mainly because spaces are reserved, and that will be for separating arguments. 00:01:33.380 --> 00:01:36.700 So, for example, something like foo equals bar 00:01:36.700 --> 00:01:42.000 won't work, and the shell is gonna tell you why it's not working. 00:01:42.000 --> 00:01:46.280 It's because the foo command is not working, like foo is non-existent. 00:01:46.280 --> 00:01:47.780 And here what is actually happening, we're not assigning foo to bar, 00:01:47.780 --> 00:01:52.260 what is happening is we're calling the foo program 00:01:52.260 --> 00:01:57.520 with the first argument "=" and the second argument "bar". 00:01:57.520 --> 00:02:03.880 And in general, whenever you are having some issues, like some files with spaces 00:02:03.880 --> 00:02:06.160 you will need to be careful about that. 00:02:06.160 --> 00:02:10.620 You need to be careful about quoting strings. 00:02:10.640 --> 00:02:16.480 So, going into that, how you do strings in bash. There are two ways that you can define a string: 00:02:16.540 --> 00:02:24.720 You can define strings using double quotes and you can define strings using single, 00:02:24.720 --> 00:02:26.540 sorry, 00:02:26.540 --> 00:02:28.880 using single quotes. 00:02:29.140 --> 00:02:32.760 However, for literal strings they are equivalent, 00:02:32.760 --> 00:02:35.460 but for the rest they are not equivalent. 00:02:35.460 --> 00:02:42.980 So, for example, if we do value is $foo, 00:02:43.440 --> 00:02:48.480 the $foo has been expanded like a string, substituted to the 00:02:48.480 --> 00:02:50.820 value of the foo variable in the shell. 00:02:50.960 --> 00:02:58.940 Whereas if we do this with a simple quote, we are just getting the $foo as it is 00:02:58.940 --> 00:03:02.280 and single quotes won't be replacing. Again, 00:03:02.280 --> 00:03:07.290 it's really easy to write a script, assume that this is kind of like Python, that you might be 00:03:07.290 --> 00:03:10.860 more familiar with, and not realize all that. 00:03:10.860 --> 00:03:14.180 And this is the way you will assign variables. 00:03:14.180 --> 00:03:17.849 Then bash also has control flow techniques that we'll see later, 00:03:17.849 --> 00:03:24.440 like for loops, while loops, and one main thing is you can define functions. 00:03:24.440 --> 00:03:27.820 We can access a function I have defined here. 00:03:28.220 --> 00:03:34.220 Here we have the MCD function, that has been defined, and the thing is 00:03:34.220 --> 00:03:38.400 so far, we have just kind of seen how to execute several commands by piping 00:03:38.400 --> 00:03:40.720 into them, kind of saw that briefly yesterday. 00:03:40.940 --> 00:03:44.980 But a lot of times you want to do first one thing and then another thing. 00:03:44.980 --> 00:03:47.580 And that's kind of like the 00:03:47.740 --> 00:03:50.880 sequential execution that we get here. 00:03:50.880 --> 00:03:54.260 Here, for example, we're calling the MCD function. 00:03:56.860 --> 00:03:57.800 We, first, 00:03:57.800 --> 00:04:02.960 are calling the makedir command, which is creating this directory. 00:04:02.960 --> 00:04:05.600 Here, $1 is like a special variable. 00:04:05.600 --> 00:04:07.440 This is the way that bash works, 00:04:07.440 --> 00:04:12.160 whereas in other scripting languages there will be like argv, 00:04:12.160 --> 00:04:16.620 the first item of the array argv will contain the argument. 00:04:16.620 --> 00:04:19.160 In bash it's $1. And in general, a lot 00:04:19.160 --> 00:04:21.640 of things in bash will be dollar something 00:04:21.640 --> 00:04:26.680 and will be reserved, we will be seeing more examples later. 00:04:26.680 --> 00:04:30.290 And once we have created the folder, we CD into that folder, 00:04:30.290 --> 00:04:34.687 which is kind of a fairly common pattern that you will see. 00:04:34.687 --> 00:04:39.060 We will actually type this directly into our shell, and it will work and 00:04:39.120 --> 00:04:45.260 it will define this function. But sometimes it's nicer to write things in a file. 00:04:45.260 --> 00:04:50.040 What we can do is we can source this. And that will 00:04:50.080 --> 00:04:53.960 execute this script in our shell and load it. 00:04:53.960 --> 00:04:59.340 So now it looks like nothing happened, but now the MCD function has 00:04:59.340 --> 00:05:03.460 been defined in our shell. So we can now for example do 00:05:03.463 --> 00:05:09.150 MCD test, and now we move from the tools directory to the test 00:05:09.160 --> 00:05:14.200 directory. We both created the folder and we moved into it. 00:05:15.760 --> 00:05:18.820 What else. So a result is... 00:05:18.820 --> 00:05:22.160 We can access the first argument with $1. 00:05:22.160 --> 00:05:26.100 There's a lot more reserved commands, 00:05:26.100 --> 00:05:30.020 for example $0 will be the name of the script, 00:05:30.020 --> 00:05:35.260 $2 through $9 will be the second through the ninth arguments 00:05:35.260 --> 00:05:38.070 that the bash script takes. Some of these reserved 00:05:38.070 --> 00:05:43.080 keywords can be directly used in the shell, so for example 00:05:43.420 --> 00:05:50.300 $? will get you the error code from the previous command, 00:05:50.300 --> 00:05:53.580 which I'll also explain briefly. 00:05:53.580 --> 00:05:58.320 But for example, $_ will get you the last argument of the 00:05:58.320 --> 00:06:03.460 previous command. So another way we could have done this is 00:06:03.460 --> 00:06:07.380 we could have said like "mkdir test" 00:06:07.380 --> 00:06:12.020 and instead of rewriting test, we can access that last argument 00:06:12.020 --> 00:06:18.400 as part of the (previous command), using $_ 00:06:18.400 --> 00:06:23.160 like, that will be replaced with test and now we go into test. 00:06:25.040 --> 00:06:27.480 There are a lot of them, you should familiarize with them. 00:06:27.480 --> 00:06:32.900 Another one I often use is called "bang bang" ("!!"), you will run into this 00:06:32.910 --> 00:06:37.300 whenever you, for example, are trying to create something and you don't have 00:06:37.320 --> 00:06:41.000 enough permissions. Then, you can do "sudo !!" 00:06:41.010 --> 00:06:43.400 and then that will replace the command in 00:06:43.470 --> 00:06:46.400 there and now you can just try doing 00:06:46.440 --> 00:06:48.380 that. And now it will prompt you for a password, 00:06:48.380 --> 00:06:50.080 because you have sudo permissions. 00:06:53.800 --> 00:06:57.180 Before, I mentioned the, kind of the error command. 00:06:57.180 --> 00:06:59.400 Yesterday we saw that, in general, there are 00:06:59.400 --> 00:07:02.400 different ways a process can communicate 00:07:02.400 --> 00:07:05.091 with other processes or commands. 00:07:05.100 --> 00:07:08.420 We mentioned the standard input, which also was like 00:07:09.160 --> 00:07:11.380 getting stuff through the standard input, 00:07:11.640 --> 00:07:13.840 putting stuff into the standard output. 00:07:13.840 --> 00:07:16.830 There are a couple more interesting things, there's also like a 00:07:16.830 --> 00:07:19.837 standard error, a stream where you write errors 00:07:19.837 --> 00:07:23.900 that happen with your program and you don't want to pollute the standard output. 00:07:23.900 --> 00:07:27.420 There's also the error code, which is like a general 00:07:27.420 --> 00:07:29.520 thing in a lot of programming languages, 00:07:29.520 --> 00:07:34.460 some way of reporting how the entire run of something went. 00:07:34.460 --> 00:07:36.060 So if we do 00:07:36.060 --> 00:07:41.020 something like echo hello and we 00:07:41.580 --> 00:07:43.920 query for the value, it's zero. And it's zero 00:07:43.920 --> 00:07:45.840 because everything went okay and there 00:07:45.840 --> 00:07:49.170 weren't any issues. And a zero exit code is 00:07:49.170 --> 00:07:50.940 the same as you will get in a language 00:07:50.940 --> 00:07:54.980 like C, like 0 means everything went fine, there were no errors. 00:07:54.980 --> 00:07:57.600 However, sometimes things won't work. 00:07:57.600 --> 00:08:04.600 Sometimes, like if we try to grep for foobar in our MCD script, 00:08:04.600 --> 00:08:08.130 and now we check for that value, it's 1. And that's 00:08:08.130 --> 00:08:10.770 because we tried to search for the foobar 00:08:10.770 --> 00:08:13.620 string in the MCD script and it wasn't there. 00:08:13.620 --> 00:08:17.190 So grep doesn't print anything, but 00:08:17.190 --> 00:08:19.950 let us know that things didn't work by 00:08:19.950 --> 00:08:22.260 giving us a 1 error code. 00:08:22.260 --> 00:08:24.420 There are some interesting commands like 00:08:24.420 --> 00:08:29.160 "true", for example, will always have a zero 00:08:29.160 --> 00:08:35.060 error code, and false will always have a one error code. 00:08:35.060 --> 00:08:37.919 Then there are like 00:08:37.919 --> 00:08:40.080 these logical operators that you can use 00:08:40.080 --> 00:08:43.808 to do some sort of conditionals. For example, one way... 00:08:43.808 --> 00:08:47.160 you also have IF's and ELSE's, that we will see later, but you can do 00:08:47.160 --> 00:08:51.920 something like "false", and echo "Oops fail". 00:08:51.920 --> 00:08:56.300 So here we have two commands connected by this OR operator. 00:08:56.300 --> 00:09:00.250 What bash is gonna do here, it's gonna execute the first one 00:09:00.250 --> 00:09:04.450 and if the first one didn't work, then it's 00:09:04.450 --> 00:09:07.380 gonna execute the second one. So here we get it, 00:09:07.380 --> 00:09:12.000 because it's gonna try to do a logical OR. If the first one didn't have 00:09:12.000 --> 00:09:15.960 a zero error code, it's gonna try to do the second one. Similarly, if we 00:09:15.960 --> 00:09:19.580 instead of use "false", we use something like "true", 00:09:19.580 --> 00:09:22.180 since true will have a zero error code, then the 00:09:22.180 --> 00:09:24.700 second one will be short-circuited and 00:09:24.700 --> 00:09:27.500 it won't be printed. 00:09:32.560 --> 00:09:36.970 Similarly, we have an AND operator which will only 00:09:36.970 --> 00:09:39.430 execute the second part if the first one 00:09:39.430 --> 00:09:41.440 ran without errors. 00:09:41.440 --> 00:09:44.820 And the same thing will happen. 00:09:44.820 --> 00:09:50.340 If the first one fails, then the second part of this thing won't be executed. 00:09:50.340 --> 00:09:57.280 Kind of not exactly related to that, but another thing that you will see is 00:10:00.020 --> 00:10:04.120 that no matter what you execute, then you can concatenate 00:10:04.120 --> 00:10:07.120 commands using a semicolon in the same line, 00:10:07.120 --> 00:10:10.300 and that will always print. 00:10:10.300 --> 00:10:13.630 Beyond that, what we haven't seen, for example, is how 00:10:13.630 --> 00:10:19.460 you go about getting the output of a command into a variable. 00:10:19.630 --> 00:10:24.120 And the way we can do that is doing something like this. 00:10:24.120 --> 00:10:29.480 What we're doing here is we're getting the output of the PWD command, 00:10:29.480 --> 00:10:32.720 which is just printing the present working directory 00:10:32.720 --> 00:10:33.740 where we are right now. 00:10:33.740 --> 00:10:37.220 And then we're storing that into the foo variable. 00:10:37.220 --> 00:10:42.279 So we do that and then we ask for foo, we view our string. 00:10:42.280 --> 00:10:48.460 More generally, we can do this thing called command substitution 00:10:50.110 --> 00:10:51.500 by putting it into any string. 00:10:51.500 --> 00:10:55.162 And since we're using double quotes instead of single quotes 00:10:55.162 --> 00:10:57.440 that thing will be expanded and 00:10:57.440 --> 00:11:02.740 it will tell us that we are in this working folder. 00:11:02.740 --> 00:11:09.240 Another interesting thing is, right now, what this is expanding to is a string 00:11:09.400 --> 00:11:10.300 instead of 00:11:11.920 --> 00:11:13.320 It's just expanding as a string. 00:11:13.460 --> 00:11:17.640 Another nifty and lesser known tool is called process substitution, 00:11:17.640 --> 00:11:20.540 which is kind of similar. What it will do... 00:11:24.360 --> 00:11:30.041 it will, here for example, the "<(", some command and another parenthesis, 00:11:30.041 --> 00:11:34.840 what that will do is: that will execute, that will get the output to 00:11:34.840 --> 00:11:39.120 kind of like a temporary file and it will give the file handle to the command. 00:11:39.120 --> 00:11:42.020 So here what we're doing is we're getting... 00:11:42.020 --> 00:11:45.760 we're LS'ing the directory, putting it into a temporary file, 00:11:45.760 --> 00:11:48.040 doing the same thing for the parent folder and then 00:11:48.040 --> 00:11:51.310 we're concatenating both files. And this 00:11:51.310 --> 00:11:55.520 will, may be really handy, because some commands instead of expecting 00:11:55.520 --> 00:11:59.500 the input coming from the stdin, they are expecting things to 00:11:59.500 --> 00:12:03.560 come from some file that is giving some of the arguments. 00:12:04.700 --> 00:12:07.620 So we get both things concatenated. 00:12:12.880 --> 00:12:17.040 I think so far there's been a lot of information, let's see a simple, 00:12:17.040 --> 00:12:22.920 an example script where we see a few of these things. 00:12:23.200 --> 00:12:27.220 So for example here we have a string and we 00:12:27.220 --> 00:12:30.327 have this $date. So $date is a program. 00:12:30.327 --> 00:12:34.540 Again there's a lot of programs in UNIX you will kind of slowly 00:12:34.540 --> 00:12:36.120 familiarize with a lot of them. 00:12:36.120 --> 00:12:42.820 Date just prints what the current date is and you can specify different formats. 00:12:43.800 --> 00:12:48.700 Then, we have these $0 here. $0 is the name 00:12:48.700 --> 00:12:50.540 of the script that we're running. 00:12:50.550 --> 00:12:56.590 Then we have $#, that's the number of arguments that we are giving 00:12:56.590 --> 00:13:01.920 to the command, and then $$ is the process ID of this command that is running. 00:13:01.920 --> 00:13:06.160 Again, there's a lot of these dollar things, they're not intuitive 00:13:06.160 --> 00:13:07.690 because they don't have like a mnemonic 00:13:07.690 --> 00:13:10.450 way of remembering, maybe, $#. But 00:13:10.450 --> 00:13:12.880 it can be... you will just be 00:13:12.880 --> 00:13:14.660 seeing them and getting familiar with them. 00:13:14.660 --> 00:13:19.200 Here we have this $@, and that will expand to all the arguments. 00:13:19.200 --> 00:13:21.480 So, instead of having to assume that, 00:13:21.490 --> 00:13:25.840 maybe say, we have three arguments and writing $1, $2, $3, 00:13:25.840 --> 00:13:29.760 if we don't know how many arguments we can put all those arguments there. 00:13:29.760 --> 00:13:33.670 And that has been given to a for loop. And the for loop 00:13:33.670 --> 00:13:39.020 will, in time, get the file variable 00:13:39.020 --> 00:13:43.880 and it will be giving each one of the arguments. 00:13:43.880 --> 00:13:47.529 So what we're doing is, for every one of the arguments we're giving. 00:13:47.529 --> 00:13:51.699 Then, in the next line we're running the 00:13:51.699 --> 00:13:56.920 grep command which is just search for a substring in some file and we're 00:13:56.920 --> 00:14:01.380 searching for the string foobar in the file. 00:14:01.380 --> 00:14:06.490 Here, we have put the variable that the file took, to expand. 00:14:06.490 --> 00:14:11.559 And yesterday we saw that if we care about the output of a program, we can 00:14:11.560 --> 00:14:15.680 redirect it to somewhere, to save it or to connect it to some other file. 00:14:15.680 --> 00:14:18.939 But sometimes you want the opposite. 00:14:18.939 --> 00:14:21.260 Sometimes, here for example, we care... 00:14:21.260 --> 00:14:25.119 we're gonna care about the error code. About this script, we're gonna care whether the 00:14:25.120 --> 00:14:28.440 grep ran successfully or it didn't. 00:14:28.440 --> 00:14:33.220 So we can actually discard entirely what the output... 00:14:33.220 --> 00:14:37.480 like both the standard output and the standard error of the grep command. 00:14:37.480 --> 00:14:39.970 And what we're doing is we're 00:14:39.970 --> 00:14:43.029 redirecting the output to /dev/null which 00:14:43.029 --> 00:14:46.540 is kind of like a special device in UNIX 00:14:46.540 --> 00:14:49.119 systems where you can like write and 00:14:49.119 --> 00:14:51.129 it will be discarded. Like you can 00:14:51.129 --> 00:14:52.869 write no matter how much you want, 00:14:52.869 --> 00:14:57.730 there, and it will be discarded. And here's the ">" symbol 00:14:57.730 --> 00:15:02.199 that we saw yesterday for redirecting output. Here you have a "2>" 00:15:02.199 --> 00:15:04.689 and, as some of you might have 00:15:04.689 --> 00:15:06.519 guessed by now, this is for redirecting the 00:15:06.519 --> 00:15:08.589 standard error, because those those two 00:15:08.589 --> 00:15:11.709 streams are separate, and you kind of have to 00:15:11.709 --> 00:15:14.639 tell bash what to do with each one of them. 00:15:14.639 --> 00:15:17.529 So here, we run, we check if the file has 00:15:17.529 --> 00:15:20.649 foobar, and if the file has foobar then it's 00:15:20.649 --> 00:15:22.959 going to have a zero code. If it 00:15:22.959 --> 00:15:24.369 doesn't have foobar, it's gonna have a 00:15:24.369 --> 00:15:26.980 nonzero error code. So that's exactly what we 00:15:26.980 --> 00:15:31.120 check. In this if part of the command we 00:15:31.120 --> 00:15:34.840 say "get me the error code". Again, this $? 00:15:34.840 --> 00:15:37.240 And then we have a comparison operator 00:15:37.240 --> 00:15:41.590 which is "-ne", for "non equal". And some 00:15:41.590 --> 00:15:47.650 other programming languages will have "==", "!=", these 00:15:47.650 --> 00:15:51.070 symbols. In bash there's 00:15:51.070 --> 00:15:53.650 like a reserved set of comparisons and 00:15:53.650 --> 00:15:54.970 it's mainly because there's a lot of 00:15:54.970 --> 00:15:57.520 things you might want to test for when 00:15:57.520 --> 00:15:59.080 you're in the shell. Here for example 00:15:59.080 --> 00:16:03.970 we're just checking for two values, two integer values, being the same. Or for 00:16:03.970 --> 00:16:08.380 example here, the "-F" check will let 00:16:08.380 --> 00:16:10.420 us know if a file exists, which is 00:16:10.420 --> 00:16:12.220 something that you will run into very, 00:16:12.220 --> 00:16:17.530 very commonly. I'm going back to the 00:16:17.530 --> 00:16:23.020 example. Then, what happens when we 00:16:24.400 --> 00:16:28.600 if the file did not have foobar, like there was a 00:16:28.600 --> 00:16:31.990 nonzero error code, then we print 00:16:31.990 --> 00:16:33.400 "this file doesn't have any foobar, 00:16:33.400 --> 00:16:36.400 we're going to add one". And what we do is 00:16:36.400 --> 00:16:40.750 we echo this "# foobar", hoping this 00:16:40.750 --> 00:16:43.200 is a comment to the file and then we're 00:16:43.200 --> 00:16:47.620 using the operator ">>" to append at the end of 00:16:47.620 --> 00:16:50.800 the file. Here since the file has 00:16:50.800 --> 00:16:54.490 been fed through the script, and we don't know it beforehand, we have to substitute 00:16:54.490 --> 00:17:03.430 the variable of the filename. We can actually run this. We already have 00:17:03.430 --> 00:17:05.260 correct permissions in this script and 00:17:05.260 --> 00:17:10.540 we can give a few examples. We have a few files in this folder, "mcd" is the 00:17:10.540 --> 00:17:12.760 one we saw at the beginning for the MCD 00:17:12.760 --> 00:17:15.040 function, some other "script" function and 00:17:15.040 --> 00:17:21.700 we can even feed the own script to itself to check if it has foobar in it. 00:17:21.700 --> 00:17:26.680 And we run it and first we can see that there's different 00:17:26.680 --> 00:17:29.460 variables that we saw, that have been 00:17:29.460 --> 00:17:33.400 successfully expanded. We have the date, that has 00:17:33.400 --> 00:17:36.700 been replaced to the current time, then 00:17:36.700 --> 00:17:39.100 we're running this program, with three 00:17:39.100 --> 00:17:44.560 arguments, this randomized PID, and then 00:17:44.560 --> 00:17:46.510 it's telling us MCD doesn't have any 00:17:46.510 --> 00:17:48.169 foobar, so we are adding a new one, 00:17:48.169 --> 00:17:50.450 and this script file doesn't 00:17:50.450 --> 00:17:52.970 have one. So now for example let's look at MCD 00:17:52.970 --> 00:17:55.820 and it has the comment that we were looking for. 00:17:59.000 --> 00:18:05.619 One other thing to know when you're executing scripts is that 00:18:05.619 --> 00:18:07.759 here we have like three completely 00:18:07.759 --> 00:18:10.279 different arguments but very commonly 00:18:10.279 --> 00:18:12.889 you will be giving arguments that 00:18:12.889 --> 00:18:16.100 can be more succinctly given in some way. 00:18:16.100 --> 00:18:20.179 So for example here if we wanted to 00:18:20.179 --> 00:18:25.429 refer to all the ".sh" scripts we 00:18:25.429 --> 00:18:31.120 could just do something like "ls *.sh" 00:18:31.120 --> 00:18:36.120 and this is a way of filename expansion that most shells have 00:18:36.120 --> 00:18:38.450 that's called "globbing". Here, as you 00:18:38.450 --> 00:18:39.919 might expect, this is gonna say 00:18:39.919 --> 00:18:42.559 anything that has any kind of sort of 00:18:42.559 --> 00:18:45.940 characters and ends up with "sh". 00:18:45.940 --> 00:18:52.159 Unsurprisingly, we get "example.sh" and "mcd.sh". We also have these 00:18:52.159 --> 00:18:54.769 "project1" and "project2", and if there 00:18:54.769 --> 00:19:00.100 were like a... we can do a "project42", for example 00:19:00.620 --> 00:19:04.220 And now if we just want to refer to the projects that have 00:19:04.220 --> 00:19:07.279 a single character, but not two characters 00:19:07.279 --> 00:19:08.720 afterwards, like any other characters, 00:19:08.720 --> 00:19:13.879 we can use the question mark. So "?" will expand to only a single one. 00:19:13.880 --> 00:19:17.360 And we get, LS'ing, first 00:19:17.360 --> 00:19:21.049 "project1" and then "project2". 00:19:21.049 --> 00:19:27.580 In general, globbing can be very powerful. You can also combine it. 00:19:31.880 --> 00:19:35.480 A common pattern is to use what is called curly braces. 00:19:35.480 --> 00:19:39.320 So let's say we have an image, that we have in this folder 00:19:39.320 --> 00:19:43.620 and we want to convert this image from PNG to JPG 00:19:43.620 --> 00:19:46.320 or we could maybe copy it, or... 00:19:46.320 --> 00:19:49.609 it's a really common pattern, to have two or more arguments that are 00:19:49.609 --> 00:19:55.240 fairly similar and you want to do something with them as arguments to some command. 00:19:55.240 --> 00:20:01.290 You could do it this way, or more succinctly, you can just do 00:20:01.290 --> 00:20:08.880 "image.{png,jpg}" 00:20:09.410 --> 00:20:13.590 And here, I'm getting some color feedback, but what this will do, is 00:20:13.590 --> 00:20:17.610 it'll expand into the line above. 00:20:17.610 --> 00:20:23.990 Actually, I can ask zsh to do that for me. And that what's happening here. 00:20:23.990 --> 00:20:26.550 This is really powerful. So for example 00:20:26.550 --> 00:20:29.220 you can do something like... we could do... 00:20:29.220 --> 00:20:34.220 "touch" on a bunch of foo's, and all of this will be expanded. 00:20:35.520 --> 00:20:41.880 You can also do it at several levels and you will do the Cartesian... 00:20:41.880 --> 00:20:49.980 if we have something like this, we have one group here, "{1,2}" 00:20:49.980 --> 00:20:53.310 and then here there's "{1,2,3}", and this is going to do 00:20:53.310 --> 00:20:54.990 the Cartesian product of these 00:20:54.990 --> 00:20:59.920 two expansions and it will expand into all these things, 00:20:59.960 --> 00:21:03.540 that we can quickly "touch". 00:21:03.540 --> 00:21:10.520 You can also combine the asterisk glob with the curly braces glob. 00:21:10.520 --> 00:21:16.840 You can even use kind of ranges. Like, we can do "mkdir" 00:21:16.840 --> 00:21:21.420 and we create the "foo" and the "bar" directories, and then we 00:21:21.420 --> 00:21:25.680 can do something along these lines. This 00:21:25.680 --> 00:21:28.890 is going to expand to "fooa", "foob"... 00:21:28.890 --> 00:21:31.430 like all these combinations, through "j", and 00:21:31.430 --> 00:21:35.250 then the same for "bar". I haven't 00:21:35.250 --> 00:21:38.610 really tested it... but yeah, we're getting all these combinations that we 00:21:38.610 --> 00:21:41.850 can "touch". And now, if we touch something 00:21:41.850 --> 00:21:47.970 that is different between these two [directories], we 00:21:47.970 --> 00:21:55.890 can again showcase the process substitution that we saw 00:21:55.890 --> 00:21:59.610 earlier. Say we want to check what files are different between these 00:21:59.610 --> 00:22:03.400 two folders. For us it's obvious, we just saw it, it's X and Y, 00:22:03.400 --> 00:22:07.410 but we can ask the shell to do this "diff" for us between the 00:22:07.410 --> 00:22:10.200 output of one LS and the other LS. 00:22:10.200 --> 00:22:12.810 Unsurprisingly we're getting: X is 00:22:12.810 --> 00:22:14.700 only in the first folder and Y is 00:22:14.700 --> 00:22:20.970 only in the second folder. What is more 00:22:20.970 --> 00:22:26.519 is, right now, we have only seen bash scripts. If you like other 00:22:26.520 --> 00:22:30.260 scripts, like for some tasks bash is probably not the best, 00:22:30.260 --> 00:22:33.119 it can be tricky. You can actually write scripts that 00:22:33.119 --> 00:22:35.700 interact with the shell implemented in a lot 00:22:35.700 --> 00:22:39.710 of different languages. So for example, let's see here a 00:22:39.710 --> 00:22:43.139 Python script that has a magic line at the 00:22:43.139 --> 00:22:45.539 beginning that I'm not explaining for now. 00:22:45.540 --> 00:22:48.330 Then we have "import sys", 00:22:48.330 --> 00:22:53.629 it's kind of like... Python is not, by default, trying to interact 00:22:53.629 --> 00:22:56.999 with the shell so you will have to import 00:22:56.999 --> 00:22:58.799 some library. And then we're doing a 00:22:58.799 --> 00:23:01.529 really silly thing of just iterating 00:23:01.529 --> 00:23:06.440 over "sys.argv[1:]". 00:23:06.440 --> 00:23:12.809 "sys.argv" is kind of similar to what in bash we're getting as $0, $1, &c. 00:23:12.809 --> 00:23:16.649 Like the vector of the arguments, we're printing it in the reversed order. 00:23:16.649 --> 00:23:21.179 And the magic line at the beginning is 00:23:21.179 --> 00:23:23.999 called a shebang and is the way that the 00:23:23.999 --> 00:23:26.159 shell will know how to run this program. 00:23:26.159 --> 00:23:30.509 You can always do something like 00:23:30.509 --> 00:23:34.379 "python script.py", and then "a b c" and that 00:23:34.379 --> 00:23:36.659 will work, always, like that. But 00:23:36.659 --> 00:23:39.119 what if we want to make this to be 00:23:39.119 --> 00:23:41.309 executable from the shell? The way the 00:23:41.309 --> 00:23:44.190 shell knows that it has to use python as the 00:23:44.190 --> 00:23:48.450 interpreter to run this file is using 00:23:48.450 --> 00:23:52.440 that first line. And that first line is 00:23:52.440 --> 00:23:56.620 giving it the path to where that thing lives. 00:23:58.500 --> 00:23:59.600 However, you might not know. 00:23:59.609 --> 00:24:01.830 Like, different machines will have probably 00:24:01.830 --> 00:24:04.049 different places where they put python 00:24:04.049 --> 00:24:06.090 and you might not want to assume where 00:24:06.090 --> 00:24:08.789 python is installed, or any other interpreter. 00:24:08.789 --> 00:24:16.379 So one thing that you can do is use the 00:24:16.380 --> 00:24:17.720 "env" command. 00:24:18.280 --> 00:24:21.560 You can also give arguments in the shebang, so 00:24:21.570 --> 00:24:23.940 what we're doing here is specifying 00:24:23.940 --> 00:24:29.720 run the "env" command, that is for pretty much every system, there are some exceptions, but like for 00:24:29.720 --> 00:24:31.550 pretty much every system it's is in 00:24:31.550 --> 00:24:33.620 "usr/bin", where a lot of binaries live, 00:24:33.620 --> 00:24:36.200 and then we're calling it with the 00:24:36.200 --> 00:24:38.570 argument "python". And then that will make 00:24:38.570 --> 00:24:42.020 use of the path environment variable 00:24:42.020 --> 00:24:43.580 that we saw in the first lecture. It's 00:24:43.580 --> 00:24:45.680 gonna search in that path for the Python 00:24:45.680 --> 00:24:48.620 binary and then it's gonna use that to 00:24:48.620 --> 00:24:50.480 interpret this file. And that will make 00:24:50.480 --> 00:24:52.490 this more portable so it can be run in 00:24:52.490 --> 00:24:57.520 my machine, and your machine and some other machine. 00:25:08.020 --> 00:25:12.140 Another thing is that the bash is not 00:25:12.140 --> 00:25:14.300 really like modern, it was 00:25:14.300 --> 00:25:16.340 developed a while ago. And sometimes 00:25:16.340 --> 00:25:18.890 it can be tricky to debug. By 00:25:18.890 --> 00:25:21.980 default, and the ways it will fail 00:25:21.980 --> 00:25:24.020 sometimes are intuitive like the way we 00:25:24.020 --> 00:25:26.180 saw before of like foo command not 00:25:26.180 --> 00:25:28.610 existing, sometimes it's not. So there's 00:25:28.610 --> 00:25:31.280 like a really nifty tool that we have 00:25:31.280 --> 00:25:34.310 linked in the lecture notes, which is called 00:25:34.310 --> 00:25:37.580 "shellcheck", that will kind of give you 00:25:37.580 --> 00:25:40.010 both warnings and syntactic errors 00:25:40.010 --> 00:25:43.250 and other things that you might not have quoted properly, 00:25:43.250 --> 00:25:46.040 or you might have misplaced spaces in 00:25:46.040 --> 00:25:50.060 your files. So for example for extremely simple "mcd.sh" 00:25:50.060 --> 00:25:51.980 file we're getting a couple 00:25:51.980 --> 00:25:54.800 of errors saying hey, surprisingly, 00:25:54.800 --> 00:25:56.090 we're missing a shebang, like this 00:25:56.090 --> 00:25:59.060 might not interpret it correctly if you're 00:25:59.060 --> 00:26:02.000 it at a different system. Also, this 00:26:02.000 --> 00:26:05.620 CD is taking a command and it might not 00:26:05.620 --> 00:26:08.960 expand properly so instead of using CD 00:26:08.960 --> 00:26:11.300 you might want to use something like CD 00:26:11.300 --> 00:26:14.540 and then an OR and then an "exit". We go 00:26:14.540 --> 00:26:16.490 back to what we explained earlier, what 00:26:16.490 --> 00:26:18.920 this will do is like if the 00:26:18.920 --> 00:26:21.860 CD doesn't end correctly, you cannot CD 00:26:21.860 --> 00:26:23.720 into the folder because either you 00:26:23.720 --> 00:26:25.250 don't have permissions, it doesn't exist... 00:26:25.250 --> 00:26:28.780 That will give a nonzero error 00:26:28.780 --> 00:26:32.420 command, so you will execute exit 00:26:32.420 --> 00:26:33.920 and that will stop the script 00:26:33.920 --> 00:26:35.810 instead of continue executing as if 00:26:35.810 --> 00:26:37.240 you were in a place that you are 00:26:37.240 --> 00:26:42.900 actually not in. And actually I haven't tested, but I 00:26:42.920 --> 00:26:47.179 think we can check for "example.sh" 00:26:47.179 --> 00:26:50.809 and here we're getting that we should be 00:26:50.809 --> 00:26:55.070 checking the exit code in a different way, because it's 00:26:55.070 --> 00:26:57.710 probably not the best way, doing it this 00:26:57.710 --> 00:27:01.580 way. One last remark I want to make 00:27:01.580 --> 00:27:05.090 is that when you're writing bash scripts 00:27:05.090 --> 00:27:07.159 or functions for that matter, 00:27:07.159 --> 00:27:09.080 there's kind of a difference between 00:27:09.080 --> 00:27:12.590 writing bash scripts in isolation like a 00:27:12.590 --> 00:27:14.149 thing that you're gonna run, and a thing 00:27:14.149 --> 00:27:16.100 that you're gonna load into your shell. 00:27:16.100 --> 00:27:19.850 We will see some of this in the command 00:27:19.850 --> 00:27:23.090 line environment lecture, where we will kind of 00:27:23.090 --> 00:27:29.059 be tooling with the bashrc and the sshrc. But in general, if you make 00:27:29.059 --> 00:27:31.370 changes to for example where you are, 00:27:31.370 --> 00:27:34.009 like if you CD into a bash script and you 00:27:34.009 --> 00:27:36.919 just execute that bash script, it won't CD 00:27:36.919 --> 00:27:39.980 into the shell are right now. But if you 00:27:39.980 --> 00:27:42.980 have loaded the code directly into 00:27:42.980 --> 00:27:45.559 your shell, for example you load... 00:27:45.559 --> 00:27:48.440 you source the function and then you execute 00:27:48.440 --> 00:27:50.269 the function then you will get those 00:27:50.269 --> 00:27:52.000 side effects. And the same goes for 00:27:52.000 --> 00:27:57.220 defining variables into the shell. 00:27:57.220 --> 00:28:03.950 Now I'm going to talk about some tools that I think are nifty when 00:28:03.950 --> 00:28:07.580 working with the shell. The first was 00:28:07.580 --> 00:28:09.799 also briefly introduced yesterday. 00:28:09.799 --> 00:28:13.309 How do you know what flags, or like 00:28:13.309 --> 00:28:15.320 what exact commands are. Like how I am 00:28:15.320 --> 00:28:21.889 supposed to know that LS minus L will list the files in a list format, or that 00:28:21.889 --> 00:28:25.789 if I do "move - i", it's gonna like prom me 00:28:25.789 --> 00:28:28.639 for stuff. For that what you have is the "man" 00:28:28.639 --> 00:28:30.730 command. And the man command will kind of 00:28:30.730 --> 00:28:33.590 have like a lot of information of how 00:28:33.590 --> 00:28:35.809 will you go about... so for example here it 00:28:35.809 --> 00:28:40.340 will explain for the "-i" flag, there are 00:28:40.340 --> 00:28:43.970 all these options you can do. That's 00:28:43.970 --> 00:28:45.620 actually pretty useful and it will work 00:28:45.620 --> 00:28:51.540 not only for really simple commands that come packaged with your OS 00:28:51.540 --> 00:28:55.809 but will also work with some tools that you install from the internet 00:28:55.809 --> 00:28:58.240 for example, if the person that did the 00:28:58.240 --> 00:29:01.390 installation made it so that the man 00:29:01.390 --> 00:29:03.399 package were also installed. So for example 00:29:03.399 --> 00:29:06.490 a tool that we're gonna cover in a bit 00:29:06.490 --> 00:29:12.370 which is called "ripgrep" and is called with RG, this didn't 00:29:12.370 --> 00:29:14.980 come with my system but it has installed 00:29:14.980 --> 00:29:17.230 its own man page and I have it here and 00:29:17.230 --> 00:29:21.700 I can access it. For some commands the 00:29:21.700 --> 00:29:25.029 man page is useful but sometimes it can be 00:29:25.029 --> 00:29:28.270 tricky to decipher because it's more 00:29:28.270 --> 00:29:30.399 kind of a documentation and a 00:29:30.399 --> 00:29:32.679 description of all the things the tool 00:29:32.679 --> 00:29:35.860 can do. Sometimes it will have 00:29:35.860 --> 00:29:37.720 examples but sometimes not, and sometimes 00:29:37.720 --> 00:29:41.620 the tool can do a lot of things so a 00:29:41.620 --> 00:29:45.250 couple of good tools that I use commonly 00:29:45.250 --> 00:29:50.289 are "convert" or "ffmpeg", which deal with images and video respectively and 00:29:50.289 --> 00:29:52.419 the man pages are like enormous. So there's 00:29:52.419 --> 00:29:54.850 one neat tool called "tldr" that 00:29:54.850 --> 00:29:58.240 you can install and you will have like 00:29:58.240 --> 00:30:02.710 some nice kind of explanatory examples 00:30:02.710 --> 00:30:05.470 of how you want to use this command. And you 00:30:05.470 --> 00:30:07.840 can always Google for this, but I find 00:30:07.840 --> 00:30:10.120 myself saving going into the 00:30:10.120 --> 00:30:12.640 browser, looking about some examples and 00:30:12.640 --> 00:30:14.919 coming back, whereas "tldr" are 00:30:14.919 --> 00:30:16.870 community contributed and 00:30:16.870 --> 00:30:19.210 they're fairly useful. Then, 00:30:19.210 --> 00:30:23.020 the one for "ffmpeg" has a lot of 00:30:23.020 --> 00:30:24.940 useful examples that are more nicely 00:30:24.940 --> 00:30:26.799 formatted (if you don't have a huge 00:30:26.799 --> 00:30:30.820 font size for recording). Or even 00:30:30.820 --> 00:30:33.250 simple commands like "tar", that have a lot 00:30:33.250 --> 00:30:35.470 of options that you are combining. So for 00:30:35.470 --> 00:30:37.840 example, here you can be combining 2, 3... 00:30:37.840 --> 00:30:41.710 different flags and it can not be 00:30:41.710 --> 00:30:43.419 obvious, when you want to combine 00:30:43.419 --> 00:30:48.429 different ones. That's how you 00:30:48.429 --> 00:30:54.850 would go about finding more about these tools. On the topic of finding, let's try 00:30:54.850 --> 00:30:58.690 learning how to find files. You can 00:30:58.690 --> 00:31:03.100 always go "ls", and like you can go like 00:31:03.100 --> 00:31:05.950 "ls project1", and 00:31:05.950 --> 00:31:08.559 keep LS'ing all the way through. But 00:31:08.559 --> 00:31:11.740 maybe, if we already know that we want 00:31:11.740 --> 00:31:15.450 to look for all the folders called 00:31:15.450 --> 00:31:19.000 "src", then there's probably a better command 00:31:19.000 --> 00:31:21.400 for doing that. And that's "find". 00:31:21.460 --> 00:31:26.679 Find is the tool that, pretty much comes with every UNIX system. And find, 00:31:26.679 --> 00:31:35.230 we're gonna give it... here we're saying we want to call find in the 00:31:35.230 --> 00:31:37.510 current folder, remember that "." stands 00:31:37.510 --> 00:31:40.149 for the current folder, and we want the 00:31:40.149 --> 00:31:46.539 name to be "src" and we want the type to be a directory. And by typing that it's 00:31:46.539 --> 00:31:49.870 gonna recursively go through the current 00:31:49.870 --> 00:31:52.330 directory and look for all these files, 00:31:52.330 --> 00:31:58.659 or folders in this case, that match this pattern. Find has a lot of useful 00:31:58.659 --> 00:32:01.840 flags. So for example, you can even test 00:32:01.840 --> 00:32:05.440 for the path to be in a way. Here we're 00:32:05.440 --> 00:32:08.230 saying we want some number of folders, 00:32:08.230 --> 00:32:09.909 we don't really care how many folders, 00:32:09.909 --> 00:32:13.179 and then we care about all the Python 00:32:13.179 --> 00:32:17.830 scripts, all the things with the extension ".py", that are within a 00:32:17.830 --> 00:32:19.899 test folder. And we're also checking, just in 00:32:19.899 --> 00:32:21.519 cases really but we're checking just 00:32:21.519 --> 00:32:24.460 that it's also a type F, which stands for 00:32:24.460 --> 00:32:28.710 file. We're getting all these files. 00:32:28.710 --> 00:32:32.169 You can also use different flags for things 00:32:32.169 --> 00:32:34.000 that are not the path or the name. 00:32:34.000 --> 00:32:38.160 You could check things that have been 00:32:38.160 --> 00:32:42.060 modified ("-mtime" is for the modification time), things that have been 00:32:42.070 --> 00:32:44.540 modified in the last day, which is gonna 00:32:44.559 --> 00:32:46.659 be pretty much everything. So this is gonna print 00:32:46.659 --> 00:32:49.029 a lot of the files we created and files 00:32:49.029 --> 00:32:51.850 that were already there. You can even 00:32:51.850 --> 00:32:54.960 use other things like size, the owner, 00:32:54.960 --> 00:32:59.080 permissions, you name it. What is even more 00:32:59.080 --> 00:33:01.870 powerful is, "find" can find stuff 00:33:01.870 --> 00:33:04.269 but it also can do stuff when you 00:33:04.269 --> 00:33:10.690 find those files. So we could look for all 00:33:10.690 --> 00:33:14.080 the files that have a TMP 00:33:14.080 --> 00:33:18.160 extension, which is a temporary extension, and 00:33:18.160 --> 00:33:22.720 then, we can tell "find" that for every one of those files, 00:33:22.720 --> 00:33:26.350 just execute the "rm" command for them. And 00:33:26.350 --> 00:33:29.050 that will just be calling "rm" with all 00:33:29.050 --> 00:33:32.350 these files. So let's first execute it 00:33:32.350 --> 00:33:35.760 without, and then we execute it with it. 00:33:35.760 --> 00:33:38.950 Again, as with the command line 00:33:38.950 --> 00:33:41.470 philosophy, it looks like nothing 00:33:41.470 --> 00:33:48.070 happened. But since we have a zero error code, something 00:33:48.070 --> 00:33:49.540 happened - just that everything went 00:33:49.540 --> 00:33:51.490 correct and everything is fine. And now, 00:33:51.490 --> 00:33:57.810 if we look for these files, they aren't there anymore. 00:33:57.810 --> 00:34:02.950 Another nice thing about the shell in general is that there are 00:34:02.950 --> 00:34:05.890 these tools, but people will keep 00:34:05.890 --> 00:34:08.230 finding new ways, so alternative 00:34:08.230 --> 00:34:12.220 ways of writing these tools. It's nice to know about it. So, for 00:34:12.220 --> 00:34:20.020 example find if you just want to match the things that end in "tmp" 00:34:20.020 --> 00:34:24.190 it can be sometimes weird to do this thing, it has a long command. 00:34:24.190 --> 00:34:27.760 There's things like "fd", 00:34:27.760 --> 00:34:32.320 for example, that is a shorter command that by default will use regex 00:34:32.320 --> 00:34:34.899 and will ignore your gitfiles, so you 00:34:34.899 --> 00:34:38.020 don't even search for them. It 00:34:38.020 --> 00:34:42.879 will color-code, it will have better Unicode support... It's nice to 00:34:42.879 --> 00:34:45.040 know about some of these tools. But, again, 00:34:45.040 --> 00:34:52.149 the main idea is that if you are aware that these tools exist, you can 00:34:52.149 --> 00:34:53.740 save yourself a lot of time from doing 00:34:53.740 --> 00:34:57.660 kind of menial and repetitive tasks. 00:34:57.660 --> 00:35:00.010 Another command to bear in mind is like 00:35:00.010 --> 00:35:01.990 "find". Some of you may be 00:35:01.990 --> 00:35:04.300 wondering, "find" is probably just 00:35:04.300 --> 00:35:06.520 actually going through a directory 00:35:06.520 --> 00:35:09.580 structure and looking for things but 00:35:09.580 --> 00:35:11.260 what if I'm doing a lot of "finds" a day? 00:35:11.260 --> 00:35:12.850 Wouldn't it be better, doing kind of 00:35:12.850 --> 00:35:18.790 a database approach and build an index first, and then use that index 00:35:18.790 --> 00:35:21.520 and update it in some way. Well, actually 00:35:21.520 --> 00:35:23.380 most Unix systems already do it and 00:35:23.380 --> 00:35:28.170 this is through the "locate" command and 00:35:28.170 --> 00:35:31.690 the way that the locate will 00:35:31.690 --> 00:35:35.470 be used... it will just look for paths in 00:35:35.470 --> 00:35:38.680 your file system that have the substring 00:35:38.680 --> 00:35:44.710 that you want. I actually don't know if it will work... Okay, it worked. Let me try to 00:35:44.710 --> 00:35:49.840 do something like "missing-semester". 00:35:51.840 --> 00:35:53.950 You're gonna take a while but 00:35:53.950 --> 00:35:56.109 it found all these files that are somewhere 00:35:56.109 --> 00:35:57.730 in my file system and since it has 00:35:57.730 --> 00:36:01.750 built an index already on them, it's much 00:36:01.750 --> 00:36:05.680 faster. And then, to keep it updated, 00:36:05.680 --> 00:36:11.980 using the "updatedb" command that is running through cron, 00:36:13.840 --> 00:36:18.490 to update this database. Finding files, again, is 00:36:18.490 --> 00:36:23.230 really useful. Sometimes you're actually concerned about, not the files themselves, 00:36:23.230 --> 00:36:26.740 but the content of the files. For that 00:36:26.740 --> 00:36:31.420 you can use the grep command that we 00:36:31.420 --> 00:36:33.880 have seen so far. So you could do 00:36:33.880 --> 00:36:37.740 something like grep foobar in MCD, it's there. 00:36:37.740 --> 00:36:43.690 What if you want to, again, recursively search through the current 00:36:43.690 --> 00:36:45.760 structure and look for more files, right? 00:36:45.760 --> 00:36:48.700 We don't want to do this manually. 00:36:48.700 --> 00:36:51.220 We could use "find", and the "-exec", but 00:36:51.220 --> 00:36:58.920 actually "grep" has the "-R" flag that will go through the entire 00:36:58.920 --> 00:37:03.609 directory, here. And it's telling us 00:37:03.609 --> 00:37:06.579 that oh we have the foobar line in example.sh 00:37:06.579 --> 00:37:09.279 at these three places and in 00:37:09.279 --> 00:37:14.589 this other two places in foobar. This can be 00:37:14.589 --> 00:37:16.900 really convenient. Mainly, the 00:37:16.900 --> 00:37:18.940 use case for this is you know you have 00:37:18.940 --> 00:37:21.910 written some code in some programming 00:37:21.910 --> 00:37:23.859 language, and you know it's somewhere in 00:37:23.859 --> 00:37:26.200 your file system but you actually don't 00:37:26.200 --> 00:37:28.599 know. But you can actually quickly search. 00:37:28.600 --> 00:37:32.980 So for example, I can quickly search 00:37:35.660 --> 00:37:40.320 for all the Python files that I have in my 00:37:40.329 --> 00:37:45.460 scratch folder where I used the request library. 00:37:45.460 --> 00:37:47.589 And if I run this, it's giving me 00:37:47.589 --> 00:37:50.890 through all these files, exactly in 00:37:50.890 --> 00:37:53.650 what line it has been found. And here 00:37:53.650 --> 00:37:56.260 instead of using grep, which is fine, 00:37:56.260 --> 00:37:58.930 you could also do this, I'm using "ripgrep", 00:37:58.930 --> 00:38:05.260 which is kind of the same idea but again trying to bring some more 00:38:05.260 --> 00:38:09.730 niceties like color coding or file 00:38:09.730 --> 00:38:16.480 processing and other things. It think it has, also, unicode support. It's also pretty 00:38:16.480 --> 00:38:22.829 fast so you are not paying like a trade-off on this being slower and 00:38:22.829 --> 00:38:25.420 there's a lot of useful flags. You 00:38:25.420 --> 00:38:27.670 can say, oh, I actually want to get some 00:38:27.670 --> 00:38:30.460 context around those results. 00:38:33.040 --> 00:38:36.400 So I want to get like five lines of context around 00:38:36.400 --> 00:38:42.819 that, so you can see where that import lives and see code around it. 00:38:42.819 --> 00:38:44.170 Here in the import it's not really useful 00:38:44.170 --> 00:38:45.819 but like if you're looking for where you 00:38:45.819 --> 00:38:49.720 use the function, for example, it will 00:38:49.720 --> 00:38:54.010 be very handy. We can also do things like 00:38:54.010 --> 00:38:59.170 we can search, for example here,. 00:38:59.170 --> 00:39:04.839 A more advanced use, we can say, 00:39:04.840 --> 00:39:11.580 "-u" is for don't ignore hidden files, sometimes 00:39:12.520 --> 00:39:16.359 you want to be ignoring hidden files, except if you want to 00:39:16.359 --> 00:39:23.500 search config files, that are by default hidden. Then, instead of printing 00:39:23.500 --> 00:39:28.400 the matches, we're asking to do something that would be kind of hard, I think, 00:39:28.400 --> 00:39:31.380 to do with grep, out of my head, which is 00:39:31.390 --> 00:39:34.569 "I want you to print all the files that 00:39:34.569 --> 00:39:37.750 don't match the pattern I'm giving you", which 00:39:37.750 --> 00:39:40.030 may be a weird thing to ask here but 00:39:40.030 --> 00:39:42.940 then we keep going... And this pattern here 00:39:42.940 --> 00:39:45.790 is a small regex which is saying 00:39:45.790 --> 00:39:48.099 at the beginning of the line I have a 00:39:48.099 --> 00:39:51.190 "#" and a "!", and that's a shebang. 00:39:51.190 --> 00:39:53.470 Like that, we're searching here for all 00:39:53.470 --> 00:39:56.650 the files that don't have a shebang 00:39:56.650 --> 00:39:59.369 and then we're giving it, here, 00:39:59.369 --> 00:40:02.470 a "-t sh" to only look for "sh" 00:40:02.470 --> 00:40:07.660 files, because maybe all your Python or text files are fine 00:40:07.660 --> 00:40:10.000 without a shebang. And here it's telling us 00:40:10.000 --> 00:40:13.020 "oh, MCD is obviously missing a shebang" 00:40:14.760 --> 00:40:16.660 We can even... It has like some 00:40:16.660 --> 00:40:19.119 nice flags, so for example if we 00:40:19.120 --> 00:40:21.360 include the "stats" flag 00:40:28.700 --> 00:40:34.119 it will get all these results but it will also tell us information about all 00:40:34.119 --> 00:40:35.410 the things that it searched. For example, 00:40:35.410 --> 00:40:40.390 the number of matches that it found, the lines, the file searched, 00:40:40.390 --> 00:40:44.040 the bytes that it printed, &c. 00:40:44.040 --> 00:40:47.160 Similar as with "fd", sometimes it's not as useful 00:40:48.400 --> 00:40:50.619 using one specific tool or another and 00:40:50.620 --> 00:40:55.780 in fact, as ripgrep, there are several other tools. Like "ack", 00:40:55.780 --> 00:40:57.700 is the original grep alternative that was 00:40:57.700 --> 00:41:00.670 written. Then the silver searcher, 00:41:00.670 --> 00:41:04.089 "ag", was another one... and they're all 00:41:04.089 --> 00:41:05.589 pretty much interchangeable so 00:41:05.589 --> 00:41:07.630 maybe you're at a system that has one and 00:41:07.630 --> 00:41:09.670 not the other, just knowing that you can 00:41:09.670 --> 00:41:12.040 use these things with these tools can be 00:41:12.040 --> 00:41:15.549 fairly useful. Lastly, I want to cover 00:41:15.549 --> 00:41:19.780 how you go about, not finding files or code, but how you go about 00:41:19.780 --> 00:41:22.540 finding commands that you already 00:41:22.540 --> 00:41:30.160 some time figured out. The first, obvious way is just using the up arrow, 00:41:30.160 --> 00:41:34.540 and slowly going through all your history, looking for these matches. 00:41:34.540 --> 00:41:36.490 This is actually not very efficient, as 00:41:36.490 --> 00:41:42.579 you probably guessed. So the bash has ways to do this more easily. 00:41:42.579 --> 00:41:44.619 There is the "history" command, that will 00:41:44.619 --> 00:41:49.180 print your history. Here I'm in zsh and it only prints some of my history, but 00:41:49.180 --> 00:41:54.069 if I say, I want you to print everything from the beginning of time, it will print 00:41:54.069 --> 00:41:58.220 everything from the beginning of whatever this history is. 00:41:58.220 --> 00:42:00.700 And since this is a lot of results, 00:42:00.700 --> 00:42:02.589 maybe we care about the ones where we 00:42:02.589 --> 00:42:08.490 use the "convert" command to go from some type of file to some other type of file. 00:42:08.490 --> 00:42:12.940 Some image, sorry. Then, we're getting all 00:42:12.940 --> 00:42:15.849 these results here, about all the ones 00:42:15.849 --> 00:42:18.120 that match this substring. 00:42:21.280 --> 00:42:24.609 Even more, pretty much all shells by default will 00:42:24.609 --> 00:42:27.130 link "Ctrl+R", the keybinding, 00:42:27.130 --> 00:42:29.680 to do backward search. Here we 00:42:29.680 --> 00:42:31.569 have backward search, where we can 00:42:31.569 --> 00:42:34.750 type "convert" and it's finding the 00:42:34.750 --> 00:42:36.609 command that we just typed. And if we just 00:42:36.609 --> 00:42:38.619 keep hitting "Ctrl+R", it will 00:42:38.619 --> 00:42:41.740 kind of go through these matches and 00:42:41.740 --> 00:42:44.260 it will let re-execute it 00:42:44.260 --> 00:42:49.240 in place. Another thing that you can do, 00:42:49.240 --> 00:42:51.069 related to that, is you can use this 00:42:51.069 --> 00:42:53.829 really nifty tool called "fzf", which is 00:42:53.829 --> 00:42:56.280 like a fuzzy finder, like it will... 00:42:57.100 --> 00:42:58.480 It will let you do kind of 00:42:58.480 --> 00:43:02.200 like an interactive grep. We could do 00:43:02.200 --> 00:43:06.369 for example this, where we can cat our 00:43:06.369 --> 00:43:10.030 example.sh command, that will print 00:43:10.030 --> 00:43:11.680 print to the standard output, and then we 00:43:11.680 --> 00:43:14.290 can pipe it through fzf. It's just getting 00:43:14.290 --> 00:43:18.490 all the lines and then we can interactively look for the 00:43:18.490 --> 00:43:21.849 string that we care about. And the nice 00:43:21.849 --> 00:43:26.349 thing about fzf is that, if you enable the default bindings, it will bind to 00:43:26.349 --> 00:43:33.670 your "Ctrl+R" shell execution and now 00:43:33.670 --> 00:43:36.490 you can quickly and dynamically like 00:43:36.490 --> 00:43:41.700 look for all the times you try to convert a favicon in your history. 00:43:42.020 --> 00:43:46.375 And it's also like fuzzy matching, whereas like by default in grep 00:43:46.375 --> 00:43:49.420 or these things you have to write a regex or some 00:43:49.420 --> 00:43:52.360 expression that will match within here. 00:43:52.360 --> 00:43:54.609 Here I'm just typing "convert" and "favicon" and 00:43:54.609 --> 00:43:57.369 it's just trying to do the best scan, 00:43:57.369 --> 00:44:01.349 doing the match in the lines it has. 00:44:01.349 --> 00:44:06.190 Lastly, a tool that probably you have already seen, that I've been using 00:44:06.190 --> 00:44:08.410 for not retyping these extremely long 00:44:08.410 --> 00:44:13.080 commands is this "history substring search", where 00:44:13.940 --> 00:44:15.660 as I type in my shell, 00:44:15.670 --> 00:44:19.630 and both F fail to mention but both face 00:44:19.630 --> 00:44:22.760 which I think was originally introduced, this concept, and then 00:44:22.760 --> 00:44:25.760 zsh has a really nice implementation) 00:44:25.760 --> 00:44:26.800 what it'll let you do is 00:44:26.800 --> 00:44:31.300 as you type the command, it will dynamically search back in your 00:44:31.300 --> 00:44:34.420 history to the same command that has a common prefix, 00:44:34.980 --> 00:44:36.900 and then, if you... 00:44:39.100 --> 00:44:42.100 it will change as the match list stops 00:44:42.100 --> 00:44:44.110 working and then as you do the 00:44:44.120 --> 00:44:49.760 right arrow you can select that command and then re-execute it. 00:45:05.800 --> 00:45:09.920 We've seen a bunch of stuff... I think I have 00:45:09.940 --> 00:45:16.180 a few minutes left so I'm going to cover a couple of tools to do 00:45:16.180 --> 00:45:20.060 really quick directory listing and directory navigation. 00:45:20.060 --> 00:45:30.020 So you can always use the "-R" to recursively list some directory structure, 00:45:30.020 --> 00:45:35.160 but that can be suboptimal, I cannot really make sense of this easily. 00:45:36.340 --> 00:45:44.460 There's tool called "tree" that will be the much more friendly form of 00:45:44.460 --> 00:45:47.500 printing all the stuff, it will also color code based on... 00:45:47.500 --> 00:45:50.680 here for example "foo" is blue because it's a directory and 00:45:50.680 --> 00:45:55.100 this is red because it has execute permissions. 00:45:55.100 --> 00:46:00.220 But we can go even further than that. There's really nice tools 00:46:00.220 --> 00:46:04.580 like a recent one called "broot" that will do the same thing but here 00:46:04.580 --> 00:46:07.300 for example instead of doing this thing of listing 00:46:07.300 --> 00:46:09.160 every single file, for example in bar 00:46:09.160 --> 00:46:11.400 we have these "a" through "j" files, 00:46:11.400 --> 00:46:14.260 it will say "oh there are more, unlisted here". 00:46:15.080 --> 00:46:18.200 I can actually start typing and it will again 00:46:18.200 --> 00:46:21.540 again facily match to the files that are there 00:46:21.540 --> 00:46:24.800 and I can quickly select them and navigate through them. 00:46:24.800 --> 00:46:28.380 So, again, it's good to know that 00:46:28.380 --> 00:46:33.340 these things exist so you don't lose a large amount of time 00:46:34.240 --> 00:46:36.180 going for these files. 00:46:37.880 --> 00:46:40.500 There are also, I think I have it installed 00:46:40.500 --> 00:46:44.829 also something more similar to what you would expect your OS to have, 00:46:44.829 --> 00:46:49.960 like Nautilus or one of the Mac finders that have like an 00:46:49.960 --> 00:46:59.260 interactive input where you can just use your navigation arrows and quickly explore. 00:46:59.260 --> 00:47:03.849 It might be overkill but you'll be surprised how quickly you can 00:47:03.849 --> 00:47:07.839 make sense of some directory structure by just navigating through it. 00:47:07.840 --> 00:47:12.780 And pretty much all of these tools will let you edit, copy files... 00:47:12.780 --> 00:47:16.880 if you just look for the options for them. 00:47:17.600 --> 00:47:20.100 The last addendum is kind of going places. 00:47:20.100 --> 00:47:24.480 We have "cd", and "cd" is nice, it will get you 00:47:26.120 --> 00:47:30.060 to a lot of places. But it's pretty handy if 00:47:30.069 --> 00:47:33.190 you can like quickly go places, 00:47:33.190 --> 00:47:36.730 either you have been to recently or that 00:47:36.730 --> 00:47:40.599 you go frequently. And you can do this in 00:47:40.599 --> 00:47:42.520 many ways there's probably... you can start 00:47:42.520 --> 00:47:44.319 thinking, oh I can make bookmarks, I can 00:47:44.319 --> 00:47:46.660 make... I can make aliases in the shell, 00:47:46.660 --> 00:47:49.020 that we will cover at some point, 00:47:49.020 --> 00:47:53.020 symlinks... But at this point, 00:47:53.020 --> 00:47:54.910 programmers have like built all these 00:47:54.910 --> 00:47:56.799 tools, so programmers have already figured 00:47:56.799 --> 00:47:59.520 out a really nice way of doing this. 00:47:59.520 --> 00:48:01.930 One way of doing this is using what is 00:48:01.930 --> 00:48:05.760 called "auto jump", which I think is not loaded here... 00:48:14.140 --> 00:48:20.100 Okay, don't worry. I will cover it in the command line environment. 00:48:21.960 --> 00:48:25.579 I think it's because I disabled the "Ctrl+R" and that also 00:48:25.579 --> 00:48:31.309 affected other parts of the script. I think at this point if anyone has 00:48:31.309 --> 00:48:35.480 any questions that are related to this, I'll be more than happy to answer 00:48:35.480 --> 00:48:37.509 them, if anything was left unclear. 00:48:37.509 --> 00:48:42.859 Otherwise, a there's a bunch of exercises that we wrote, kind of 00:48:42.859 --> 00:48:46.549 touching on these topics and we encourage you to try them and 00:48:46.549 --> 00:48:48.559 come to office hours, where we can help 00:48:48.559 --> 00:48:54.569 you figure out how to do them, or some bash quirks that are not clear.