1 00:00:00,400 --> 00:00:02,860 Okay, welcome back. 2 00:00:02,860 --> 00:00:05,920 Today we're gonna cover a couple separate 3 00:00:05,920 --> 00:00:07,620 two main topics related to the shell. 4 00:00:07,620 --> 00:00:11,240 First, we're gonna do some kind of shell scripting, mainly related to bash, 5 00:00:11,240 --> 00:00:14,160 which is the shell that most of you will start 6 00:00:14,160 --> 00:00:18,520 in Mac, or like in most Linux systems, that's the default shell. 7 00:00:18,520 --> 00:00:22,720 And it's also kind of backward compatible through other shells like zsh, it's pretty nice. 8 00:00:22,740 --> 00:00:25,940 And then we're gonna cover some other shell tools that are really convenient, 9 00:00:26,060 --> 00:00:29,320 so you avoid doing really repetitive tasks, 10 00:00:29,320 --> 00:00:31,580 like looking for some piece of code 11 00:00:31,580 --> 00:00:33,420 or for some elusive file. 12 00:00:33,420 --> 00:00:36,160 And there are already really nice built-in commands 13 00:00:36,160 --> 00:00:40,960 that will really help you to do those things. 14 00:00:40,960 --> 00:00:43,260 So yesterday we already kind of introduced 15 00:00:43,260 --> 00:00:46,160 you to the shell and some of it's quirks, 16 00:00:46,160 --> 00:00:48,720 and like how you start executing commands, 17 00:00:48,720 --> 00:00:50,600 redirecting them. 18 00:00:50,600 --> 00:00:52,400 Today, we're going to kind of cover more about 19 00:00:52,460 --> 00:00:56,120 the syntax of the variables, the control flow, 20 00:00:56,120 --> 00:00:57,720 functions of the shell. 21 00:00:57,720 --> 00:01:02,700 So for example, once you drop into a shell, say you want to 22 00:01:02,760 --> 00:01:06,360 define a variable, which is one of the first things you 23 00:01:06,360 --> 00:01:09,340 learn to do in a programming language. 24 00:01:09,340 --> 00:01:12,740 Here you could do something like foo equals bar. 25 00:01:12,860 --> 00:01:18,400 And now we can access the value of foo by doing "$foo". 26 00:01:18,460 --> 00:01:21,400 And that's bar, perfect. 27 00:01:21,400 --> 00:01:24,480 One quirk that you need to be aware of is that 28 00:01:24,480 --> 00:01:27,900 spaces are really critical when you're dealing with bash. 29 00:01:27,900 --> 00:01:33,380 Mainly because spaces are reserved, and that will be for separating arguments. 30 00:01:33,380 --> 00:01:36,700 So, for example, something like foo equals bar 31 00:01:36,700 --> 00:01:42,000 won't work, and the shell is gonna tell you why it's not working. 32 00:01:42,000 --> 00:01:46,280 It's because the foo command is not working, like foo is non-existent. 33 00:01:46,280 --> 00:01:47,780 And here what is actually happening, we're not assigning foo to bar, 34 00:01:47,780 --> 00:01:52,260 what is happening is we're calling the foo program 35 00:01:52,260 --> 00:01:57,520 with the first argument "=" and the second argument "bar". 36 00:01:57,520 --> 00:02:03,880 And in general, whenever you are having some issues, like some files with spaces 37 00:02:03,880 --> 00:02:06,160 you will need to be careful about that. 38 00:02:06,160 --> 00:02:10,620 You need to be careful about quoting strings. 39 00:02:10,640 --> 00:02:16,480 So, going into that, how you do strings in bash. There are two ways that you can define a string: 40 00:02:16,540 --> 00:02:24,720 You can define strings using double quotes and you can define strings using single, 41 00:02:24,720 --> 00:02:26,540 sorry, 42 00:02:26,540 --> 00:02:28,880 using single quotes. 43 00:02:29,140 --> 00:02:32,760 However, for literal strings they are equivalent, 44 00:02:32,760 --> 00:02:35,460 but for the rest they are not equivalent. 45 00:02:35,460 --> 00:02:42,980 So, for example, if we do value is $foo, 46 00:02:43,440 --> 00:02:48,480 the $foo has been expanded like a string, substituted to the 47 00:02:48,480 --> 00:02:50,820 value of the foo variable in the shell. 48 00:02:50,960 --> 00:02:58,940 Whereas if we do this with a simple quote, we are just getting the $foo as it is 49 00:02:58,940 --> 00:03:02,280 and single quotes won't be replacing. Again, 50 00:03:02,280 --> 00:03:07,290 it's really easy to write a script, assume that this is kind of like Python, that you might be 51 00:03:07,290 --> 00:03:10,860 more familiar with, and not realize all that. 52 00:03:10,860 --> 00:03:14,180 And this is the way you will assign variables. 53 00:03:14,180 --> 00:03:17,849 Then bash also has control flow techniques that we'll see later, 54 00:03:17,849 --> 00:03:24,440 like for loops, while loops, and one main thing is you can define functions. 55 00:03:24,440 --> 00:03:27,820 We can access a function I have defined here. 56 00:03:28,220 --> 00:03:34,220 Here we have the MCD function, that has been defined, and the thing is 57 00:03:34,220 --> 00:03:38,400 so far, we have just kind of seen how to execute several commands by piping 58 00:03:38,400 --> 00:03:40,720 into them, kind of saw that briefly yesterday. 59 00:03:40,940 --> 00:03:44,980 But a lot of times you want to do first one thing and then another thing. 60 00:03:44,980 --> 00:03:47,580 And that's kind of like the 61 00:03:47,740 --> 00:03:50,880 sequential execution that we get here. 62 00:03:50,880 --> 00:03:54,260 Here, for example, we're calling the MCD function. 63 00:03:56,860 --> 00:03:57,800 We, first, 64 00:03:57,800 --> 00:04:02,960 are calling the makedir command, which is creating this directory. 65 00:04:02,960 --> 00:04:05,600 Here, $1 is like a special variable. 66 00:04:05,600 --> 00:04:07,440 This is the way that bash works, 67 00:04:07,440 --> 00:04:12,160 whereas in other scripting languages there will be like argv, 68 00:04:12,160 --> 00:04:16,620 the first item of the array argv will contain the argument. 69 00:04:16,620 --> 00:04:19,160 In bash it's $1. And in general, a lot 70 00:04:19,160 --> 00:04:21,640 of things in bash will be dollar something 71 00:04:21,640 --> 00:04:26,680 and will be reserved, we will be seeing more examples later. 72 00:04:26,680 --> 00:04:30,290 And once we have created the folder, we CD into that folder, 73 00:04:30,290 --> 00:04:34,687 which is kind of a fairly common pattern that you will see. 74 00:04:34,687 --> 00:04:39,060 We will actually type this directly into our shell, and it will work and 75 00:04:39,120 --> 00:04:45,260 it will define this function. But sometimes it's nicer to write things in a file. 76 00:04:45,260 --> 00:04:50,040 What we can do is we can source this. And that will 77 00:04:50,080 --> 00:04:53,960 execute this script in our shell and load it. 78 00:04:53,960 --> 00:04:59,340 So now it looks like nothing happened, but now the MCD function has 79 00:04:59,340 --> 00:05:03,460 been defined in our shell. So we can now for example do 80 00:05:03,463 --> 00:05:09,150 MCD test, and now we move from the tools directory to the test 81 00:05:09,160 --> 00:05:14,200 directory. We both created the folder and we moved into it. 82 00:05:15,760 --> 00:05:18,820 What else. So a result is... 83 00:05:18,820 --> 00:05:22,160 We can access the first argument with $1. 84 00:05:22,160 --> 00:05:26,100 There's a lot more reserved commands, 85 00:05:26,100 --> 00:05:30,020 for example $0 will be the name of the script, 86 00:05:30,020 --> 00:05:35,260 $2 through $9 will be the second through the ninth arguments 87 00:05:35,260 --> 00:05:38,070 that the bash script takes. Some of these reserved 88 00:05:38,070 --> 00:05:43,080 keywords can be directly used in the shell, so for example 89 00:05:43,420 --> 00:05:50,300 $? will get you the error code from the previous command, 90 00:05:50,300 --> 00:05:53,580 which I'll also explain briefly. 91 00:05:53,580 --> 00:05:58,320 But for example, $_ will get you the last argument of the 92 00:05:58,320 --> 00:06:03,460 previous command. So another way we could have done this is 93 00:06:03,460 --> 00:06:07,380 we could have said like "mkdir test" 94 00:06:07,380 --> 00:06:12,020 and instead of rewriting test, we can access that last argument 95 00:06:12,020 --> 00:06:18,400 as part of the (previous command), using $_ 96 00:06:18,400 --> 00:06:23,160 like, that will be replaced with test and now we go into test. 97 00:06:25,040 --> 00:06:27,480 There are a lot of them, you should familiarize with them. 98 00:06:27,480 --> 00:06:32,900 Another one I often use is called "bang bang" ("!!"), you will run into this 99 00:06:32,910 --> 00:06:37,300 whenever you, for example, are trying to create something and you don't have 100 00:06:37,320 --> 00:06:41,000 enough permissions. Then, you can do "sudo !!" 101 00:06:41,010 --> 00:06:43,400 and then that will replace the command in 102 00:06:43,470 --> 00:06:46,400 there and now you can just try doing 103 00:06:46,440 --> 00:06:48,380 that. And now it will prompt you for a password, 104 00:06:48,380 --> 00:06:50,080 because you have sudo permissions. 105 00:06:53,800 --> 00:06:57,180 Before, I mentioned the, kind of the error command. 106 00:06:57,180 --> 00:06:59,400 Yesterday we saw that, in general, there are 107 00:06:59,400 --> 00:07:02,400 different ways a process can communicate 108 00:07:02,400 --> 00:07:05,091 with other processes or commands. 109 00:07:05,100 --> 00:07:08,420 We mentioned the standard input, which also was like 110 00:07:09,160 --> 00:07:11,380 getting stuff through the standard input, 111 00:07:11,640 --> 00:07:13,840 putting stuff into the standard output. 112 00:07:13,840 --> 00:07:16,830 There are a couple more interesting things, there's also like a 113 00:07:16,830 --> 00:07:19,837 standard error, a stream where you write errors 114 00:07:19,837 --> 00:07:23,900 that happen with your program and you don't want to pollute the standard output. 115 00:07:23,900 --> 00:07:27,420 There's also the error code, which is like a general 116 00:07:27,420 --> 00:07:29,520 thing in a lot of programming languages, 117 00:07:29,520 --> 00:07:34,460 some way of reporting how the entire run of something went. 118 00:07:34,460 --> 00:07:36,060 So if we do 119 00:07:36,060 --> 00:07:41,020 something like echo hello and we 120 00:07:41,580 --> 00:07:43,920 query for the value, it's zero. And it's zero 121 00:07:43,920 --> 00:07:45,840 because everything went okay and there 122 00:07:45,840 --> 00:07:49,170 weren't any issues. And a zero exit code is 123 00:07:49,170 --> 00:07:50,940 the same as you will get in a language 124 00:07:50,940 --> 00:07:54,980 like C, like 0 means everything went fine, there were no errors. 125 00:07:54,980 --> 00:07:57,600 However, sometimes things won't work. 126 00:07:57,600 --> 00:08:04,600 Sometimes, like if we try to grep for foobar in our MCD script, 127 00:08:04,600 --> 00:08:08,130 and now we check for that value, it's 1. And that's 128 00:08:08,130 --> 00:08:10,770 because we tried to search for the foobar 129 00:08:10,770 --> 00:08:13,620 string in the MCD script and it wasn't there. 130 00:08:13,620 --> 00:08:17,190 So grep doesn't print anything, but 131 00:08:17,190 --> 00:08:19,950 let us know that things didn't work by 132 00:08:19,950 --> 00:08:22,260 giving us a 1 error code. 133 00:08:22,260 --> 00:08:24,420 There are some interesting commands like 134 00:08:24,420 --> 00:08:29,160 "true", for example, will always have a zero 135 00:08:29,160 --> 00:08:35,060 error code, and false will always have a one error code. 136 00:08:35,060 --> 00:08:37,919 Then there are like 137 00:08:37,919 --> 00:08:40,080 these logical operators that you can use 138 00:08:40,080 --> 00:08:43,808 to do some sort of conditionals. For example, one way... 139 00:08:43,808 --> 00:08:47,160 you also have IF's and ELSE's, that we will see later, but you can do 140 00:08:47,160 --> 00:08:51,920 something like "false", and echo "Oops fail". 141 00:08:51,920 --> 00:08:56,300 So here we have two commands connected by this OR operator. 142 00:08:56,300 --> 00:09:00,250 What bash is gonna do here, it's gonna execute the first one 143 00:09:00,250 --> 00:09:04,450 and if the first one didn't work, then it's 144 00:09:04,450 --> 00:09:07,380 gonna execute the second one. So here we get it, 145 00:09:07,380 --> 00:09:12,000 because it's gonna try to do a logical OR. If the first one didn't have 146 00:09:12,000 --> 00:09:15,960 a zero error code, it's gonna try to do the second one. Similarly, if we 147 00:09:15,960 --> 00:09:19,580 instead of use "false", we use something like "true", 148 00:09:19,580 --> 00:09:22,180 since true will have a zero error code, then the 149 00:09:22,180 --> 00:09:24,700 second one will be short-circuited and 150 00:09:24,700 --> 00:09:27,500 it won't be printed. 151 00:09:32,560 --> 00:09:36,970 Similarly, we have an AND operator which will only 152 00:09:36,970 --> 00:09:39,430 execute the second part if the first one 153 00:09:39,430 --> 00:09:41,440 ran without errors. 154 00:09:41,440 --> 00:09:44,820 And the same thing will happen. 155 00:09:44,820 --> 00:09:50,340 If the first one fails, then the second part of this thing won't be executed. 156 00:09:50,340 --> 00:09:57,280 Kind of not exactly related to that, but another thing that you will see is 157 00:10:00,020 --> 00:10:04,120 that no matter what you execute, then you can concatenate 158 00:10:04,120 --> 00:10:07,120 commands using a semicolon in the same line, 159 00:10:07,120 --> 00:10:10,300 and that will always print. 160 00:10:10,300 --> 00:10:13,630 Beyond that, what we haven't seen, for example, is how 161 00:10:13,630 --> 00:10:19,460 you go about getting the output of a command into a variable. 162 00:10:19,630 --> 00:10:24,120 And the way we can do that is doing something like this. 163 00:10:24,120 --> 00:10:29,480 What we're doing here is we're getting the output of the PWD command, 164 00:10:29,480 --> 00:10:32,720 which is just printing the present working directory 165 00:10:32,720 --> 00:10:33,740 where we are right now. 166 00:10:33,740 --> 00:10:37,220 And then we're storing that into the foo variable. 167 00:10:37,220 --> 00:10:42,279 So we do that and then we ask for foo, we view our string. 168 00:10:42,280 --> 00:10:48,460 More generally, we can do this thing called command substitution 169 00:10:50,110 --> 00:10:51,500 by putting it into any string. 170 00:10:51,500 --> 00:10:55,162 And since we're using double quotes instead of single quotes 171 00:10:55,162 --> 00:10:57,440 that thing will be expanded and 172 00:10:57,440 --> 00:11:02,740 it will tell us that we are in this working folder. 173 00:11:02,740 --> 00:11:09,240 Another interesting thing is, right now, what this is expanding to is a string 174 00:11:09,400 --> 00:11:10,300 instead of 175 00:11:11,920 --> 00:11:13,320 It's just expanding as a string. 176 00:11:13,460 --> 00:11:17,640 Another nifty and lesser known tool is called process substitution, 177 00:11:17,640 --> 00:11:20,540 which is kind of similar. What it will do... 178 00:11:24,360 --> 00:11:30,041 it will, here for example, the "<(", some command and another parenthesis, 179 00:11:30,041 --> 00:11:34,840 what that will do is: that will execute, that will get the output to 180 00:11:34,840 --> 00:11:39,120 kind of like a temporary file and it will give the file handle to the command. 181 00:11:39,120 --> 00:11:42,020 So here what we're doing is we're getting... 182 00:11:42,020 --> 00:11:45,760 we're LS'ing the directory, putting it into a temporary file, 183 00:11:45,760 --> 00:11:48,040 doing the same thing for the parent folder and then 184 00:11:48,040 --> 00:11:51,310 we're concatenating both files. And this 185 00:11:51,310 --> 00:11:55,520 will, may be really handy, because some commands instead of expecting 186 00:11:55,520 --> 00:11:59,500 the input coming from the stdin, they are expecting things to 187 00:11:59,500 --> 00:12:03,560 come from some file that is giving some of the arguments. 188 00:12:04,700 --> 00:12:07,620 So we get both things concatenated. 189 00:12:12,880 --> 00:12:17,040 I think so far there's been a lot of information, let's see a simple, 190 00:12:17,040 --> 00:12:22,920 an example script where we see a few of these things. 191 00:12:23,200 --> 00:12:27,220 So for example here we have a string and we 192 00:12:27,220 --> 00:12:30,327 have this $date. So $date is a program. 193 00:12:30,327 --> 00:12:34,540 Again there's a lot of programs in UNIX you will kind of slowly 194 00:12:34,540 --> 00:12:36,120 familiarize with a lot of them. 195 00:12:36,120 --> 00:12:42,820 Date just prints what the current date is and you can specify different formats. 196 00:12:43,800 --> 00:12:48,700 Then, we have these $0 here. $0 is the name 197 00:12:48,700 --> 00:12:50,540 of the script that we're running. 198 00:12:50,550 --> 00:12:56,590 Then we have $#, that's the number of arguments that we are giving 199 00:12:56,590 --> 00:13:01,920 to the command, and then $$ is the process ID of this command that is running. 200 00:13:01,920 --> 00:13:06,160 Again, there's a lot of these dollar things, they're not intuitive 201 00:13:06,160 --> 00:13:07,690 because they don't have like a mnemonic 202 00:13:07,690 --> 00:13:10,450 way of remembering, maybe, $#. But 203 00:13:10,450 --> 00:13:12,880 it can be... you will just be 204 00:13:12,880 --> 00:13:14,660 seeing them and getting familiar with them. 205 00:13:14,660 --> 00:13:19,200 Here we have this $@, and that will expand to all the arguments. 206 00:13:19,200 --> 00:13:21,480 So, instead of having to assume that, 207 00:13:21,490 --> 00:13:25,840 maybe say, we have three arguments and writing $1, $2, $3, 208 00:13:25,840 --> 00:13:29,760 if we don't know how many arguments we can put all those arguments there. 209 00:13:29,760 --> 00:13:33,670 And that has been given to a for loop. And the for loop 210 00:13:33,670 --> 00:13:39,020 will, in time, get the file variable 211 00:13:39,020 --> 00:13:43,880 and it will be giving each one of the arguments. 212 00:13:43,880 --> 00:13:47,529 So what we're doing is, for every one of the arguments we're giving. 213 00:13:47,529 --> 00:13:51,699 Then, in the next line we're running the 214 00:13:51,699 --> 00:13:56,920 grep command which is just search for a substring in some file and we're 215 00:13:56,920 --> 00:14:01,380 searching for the string foobar in the file. 216 00:14:01,380 --> 00:14:06,490 Here, we have put the variable that the file took, to expand. 217 00:14:06,490 --> 00:14:11,559 And yesterday we saw that if we care about the output of a program, we can 218 00:14:11,560 --> 00:14:15,680 redirect it to somewhere, to save it or to connect it to some other file. 219 00:14:15,680 --> 00:14:18,939 But sometimes you want the opposite. 220 00:14:18,939 --> 00:14:21,260 Sometimes, here for example, we care... 221 00:14:21,260 --> 00:14:25,119 we're gonna care about the error code. About this script, we're gonna care whether the 222 00:14:25,120 --> 00:14:28,440 grep ran successfully or it didn't. 223 00:14:28,440 --> 00:14:33,220 So we can actually discard entirely what the output... 224 00:14:33,220 --> 00:14:37,480 like both the standard output and the standard error of the grep command. 225 00:14:37,480 --> 00:14:39,970 And what we're doing is we're 226 00:14:39,970 --> 00:14:43,029 redirecting the output to /dev/null which 227 00:14:43,029 --> 00:14:46,540 is kind of like a special device in UNIX 228 00:14:46,540 --> 00:14:49,119 systems where you can like write and 229 00:14:49,119 --> 00:14:51,129 it will be discarded. Like you can 230 00:14:51,129 --> 00:14:52,869 write no matter how much you want, 231 00:14:52,869 --> 00:14:57,730 there, and it will be discarded. And here's the ">" symbol 232 00:14:57,730 --> 00:15:02,199 that we saw yesterday for redirecting output. Here you have a "2>" 233 00:15:02,199 --> 00:15:04,689 and, as some of you might have 234 00:15:04,689 --> 00:15:06,519 guessed by now, this is for redirecting the 235 00:15:06,519 --> 00:15:08,589 standard error, because those those two 236 00:15:08,589 --> 00:15:11,709 streams are separate, and you kind of have to 237 00:15:11,709 --> 00:15:14,639 tell bash what to do with each one of them. 238 00:15:14,639 --> 00:15:17,529 So here, we run, we check if the file has 239 00:15:17,529 --> 00:15:20,649 foobar, and if the file has foobar then it's 240 00:15:20,649 --> 00:15:22,959 going to have a zero code. If it 241 00:15:22,959 --> 00:15:24,369 doesn't have foobar, it's gonna have a 242 00:15:24,369 --> 00:15:26,980 nonzero error code. So that's exactly what we 243 00:15:26,980 --> 00:15:31,120 check. In this if part of the command we 244 00:15:31,120 --> 00:15:34,840 say "get me the error code". Again, this $? 245 00:15:34,840 --> 00:15:37,240 And then we have a comparison operator 246 00:15:37,240 --> 00:15:41,590 which is "-ne", for "non equal". And some 247 00:15:41,590 --> 00:15:47,650 other programming languages will have "==", "!=", these 248 00:15:47,650 --> 00:15:51,070 symbols. In bash there's 249 00:15:51,070 --> 00:15:53,650 like a reserved set of comparisons and 250 00:15:53,650 --> 00:15:54,970 it's mainly because there's a lot of 251 00:15:54,970 --> 00:15:57,520 things you might want to test for when 252 00:15:57,520 --> 00:15:59,080 you're in the shell. Here for example 253 00:15:59,080 --> 00:16:03,970 we're just checking for two values, two integer values, being the same. Or for 254 00:16:03,970 --> 00:16:08,380 example here, the "-F" check will let 255 00:16:08,380 --> 00:16:10,420 us know if a file exists, which is 256 00:16:10,420 --> 00:16:12,220 something that you will run into very, 257 00:16:12,220 --> 00:16:17,530 very commonly. I'm going back to the 258 00:16:17,530 --> 00:16:23,020 example. Then, what happens when we 259 00:16:24,400 --> 00:16:28,600 if the file did not have foobar, like there was a 260 00:16:28,600 --> 00:16:31,990 nonzero error code, then we print 261 00:16:31,990 --> 00:16:33,400 "this file doesn't have any foobar, 262 00:16:33,400 --> 00:16:36,400 we're going to add one". And what we do is 263 00:16:36,400 --> 00:16:40,750 we echo this "# foobar", hoping this 264 00:16:40,750 --> 00:16:43,200 is a comment to the file and then we're 265 00:16:43,200 --> 00:16:47,620 using the operator ">>" to append at the end of 266 00:16:47,620 --> 00:16:50,800 the file. Here since the file has 267 00:16:50,800 --> 00:16:54,490 been fed through the script, and we don't know it beforehand, we have to substitute 268 00:16:54,490 --> 00:17:03,430 the variable of the filename. We can actually run this. We already have 269 00:17:03,430 --> 00:17:05,260 correct permissions in this script and 270 00:17:05,260 --> 00:17:10,540 we can give a few examples. We have a few files in this folder, "mcd" is the 271 00:17:10,540 --> 00:17:12,760 one we saw at the beginning for the MCD 272 00:17:12,760 --> 00:17:15,040 function, some other "script" function and 273 00:17:15,040 --> 00:17:21,700 we can even feed the own script to itself to check if it has foobar in it. 274 00:17:21,700 --> 00:17:26,680 And we run it and first we can see that there's different 275 00:17:26,680 --> 00:17:29,460 variables that we saw, that have been 276 00:17:29,460 --> 00:17:33,400 successfully expanded. We have the date, that has 277 00:17:33,400 --> 00:17:36,700 been replaced to the current time, then 278 00:17:36,700 --> 00:17:39,100 we're running this program, with three 279 00:17:39,100 --> 00:17:44,560 arguments, this randomized PID, and then 280 00:17:44,560 --> 00:17:46,510 it's telling us MCD doesn't have any 281 00:17:46,510 --> 00:17:48,169 foobar, so we are adding a new one, 282 00:17:48,169 --> 00:17:50,450 and this script file doesn't 283 00:17:50,450 --> 00:17:52,970 have one. So now for example let's look at MCD 284 00:17:52,970 --> 00:17:55,820 and it has the comment that we were looking for. 285 00:17:59,000 --> 00:18:05,619 One other thing to know when you're executing scripts is that 286 00:18:05,619 --> 00:18:07,759 here we have like three completely 287 00:18:07,759 --> 00:18:10,279 different arguments but very commonly 288 00:18:10,279 --> 00:18:12,889 you will be giving arguments that 289 00:18:12,889 --> 00:18:16,100 can be more succinctly given in some way. 290 00:18:16,100 --> 00:18:20,179 So for example here if we wanted to 291 00:18:20,179 --> 00:18:25,429 refer to all the ".sh" scripts we 292 00:18:25,429 --> 00:18:31,120 could just do something like "ls *.sh" 293 00:18:31,120 --> 00:18:36,120 and this is a way of filename expansion that most shells have 294 00:18:36,120 --> 00:18:38,450 that's called "globbing". Here, as you 295 00:18:38,450 --> 00:18:39,919 might expect, this is gonna say 296 00:18:39,919 --> 00:18:42,559 anything that has any kind of sort of 297 00:18:42,559 --> 00:18:45,940 characters and ends up with "sh". 298 00:18:45,940 --> 00:18:52,159 Unsurprisingly, we get "example.sh" and "mcd.sh". We also have these 299 00:18:52,159 --> 00:18:54,769 "project1" and "project2", and if there 300 00:18:54,769 --> 00:19:00,100 were like a... we can do a "project42", for example 301 00:19:00,620 --> 00:19:04,220 And now if we just want to refer to the projects that have 302 00:19:04,220 --> 00:19:07,279 a single character, but not two characters 303 00:19:07,279 --> 00:19:08,720 afterwards, like any other characters, 304 00:19:08,720 --> 00:19:13,879 we can use the question mark. So "?" will expand to only a single one. 305 00:19:13,880 --> 00:19:17,360 And we get, LS'ing, first 306 00:19:17,360 --> 00:19:21,049 "project1" and then "project2". 307 00:19:21,049 --> 00:19:27,580 In general, globbing can be very powerful. You can also combine it. 308 00:19:31,880 --> 00:19:35,480 A common pattern is to use what is called curly braces. 309 00:19:35,480 --> 00:19:39,320 So let's say we have an image, that we have in this folder 310 00:19:39,320 --> 00:19:43,620 and we want to convert this image from PNG to JPG 311 00:19:43,620 --> 00:19:46,320 or we could maybe copy it, or... 312 00:19:46,320 --> 00:19:49,609 it's a really common pattern, to have two or more arguments that are 313 00:19:49,609 --> 00:19:55,240 fairly similar and you want to do something with them as arguments to some command. 314 00:19:55,240 --> 00:20:01,290 You could do it this way, or more succinctly, you can just do 315 00:20:01,290 --> 00:20:08,880 "image.{png,jpg}" 316 00:20:09,410 --> 00:20:13,590 And here, I'm getting some color feedback, but what this will do, is 317 00:20:13,590 --> 00:20:17,610 it'll expand into the line above. 318 00:20:17,610 --> 00:20:23,990 Actually, I can ask zsh to do that for me. And that what's happening here. 319 00:20:23,990 --> 00:20:26,550 This is really powerful. So for example 320 00:20:26,550 --> 00:20:29,220 you can do something like... we could do... 321 00:20:29,220 --> 00:20:34,220 "touch" on a bunch of foo's, and all of this will be expanded. 322 00:20:35,520 --> 00:20:41,880 You can also do it at several levels and you will do the Cartesian... 323 00:20:41,880 --> 00:20:49,980 if we have something like this, we have one group here, "{1,2}" 324 00:20:49,980 --> 00:20:53,310 and then here there's "{1,2,3}", and this is going to do 325 00:20:53,310 --> 00:20:54,990 the Cartesian product of these 326 00:20:54,990 --> 00:20:59,920 two expansions and it will expand into all these things, 327 00:20:59,960 --> 00:21:03,540 that we can quickly "touch". 328 00:21:03,540 --> 00:21:10,520 You can also combine the asterisk glob with the curly braces glob. 329 00:21:10,520 --> 00:21:16,840 You can even use kind of ranges. Like, we can do "mkdir" 330 00:21:16,840 --> 00:21:21,420 and we create the "foo" and the "bar" directories, and then we 331 00:21:21,420 --> 00:21:25,680 can do something along these lines. This 332 00:21:25,680 --> 00:21:28,890 is going to expand to "fooa", "foob"... 333 00:21:28,890 --> 00:21:31,430 like all these combinations, through "j", and 334 00:21:31,430 --> 00:21:35,250 then the same for "bar". I haven't 335 00:21:35,250 --> 00:21:38,610 really tested it... but yeah, we're getting all these combinations that we 336 00:21:38,610 --> 00:21:41,850 can "touch". And now, if we touch something 337 00:21:41,850 --> 00:21:47,970 that is different between these two [directories], we 338 00:21:47,970 --> 00:21:55,890 can again showcase the process substitution that we saw 339 00:21:55,890 --> 00:21:59,610 earlier. Say we want to check what files are different between these 340 00:21:59,610 --> 00:22:03,400 two folders. For us it's obvious, we just saw it, it's X and Y, 341 00:22:03,400 --> 00:22:07,410 but we can ask the shell to do this "diff" for us between the 342 00:22:07,410 --> 00:22:10,200 output of one LS and the other LS. 343 00:22:10,200 --> 00:22:12,810 Unsurprisingly we're getting: X is 344 00:22:12,810 --> 00:22:14,700 only in the first folder and Y is 345 00:22:14,700 --> 00:22:20,970 only in the second folder. What is more 346 00:22:20,970 --> 00:22:26,519 is, right now, we have only seen bash scripts. If you like other 347 00:22:26,520 --> 00:22:30,260 scripts, like for some tasks bash is probably not the best, 348 00:22:30,260 --> 00:22:33,119 it can be tricky. You can actually write scripts that 349 00:22:33,119 --> 00:22:35,700 interact with the shell implemented in a lot 350 00:22:35,700 --> 00:22:39,710 of different languages. So for example, let's see here a 351 00:22:39,710 --> 00:22:43,139 Python script that has a magic line at the 352 00:22:43,139 --> 00:22:45,539 beginning that I'm not explaining for now. 353 00:22:45,540 --> 00:22:48,330 Then we have "import sys", 354 00:22:48,330 --> 00:22:53,629 it's kind of like... Python is not, by default, trying to interact 355 00:22:53,629 --> 00:22:56,999 with the shell so you will have to import 356 00:22:56,999 --> 00:22:58,799 some library. And then we're doing a 357 00:22:58,799 --> 00:23:01,529 really silly thing of just iterating 358 00:23:01,529 --> 00:23:06,440 over "sys.argv[1:]". 359 00:23:06,440 --> 00:23:12,809 "sys.argv" is kind of similar to what in bash we're getting as $0, $1, &c. 360 00:23:12,809 --> 00:23:16,649 Like the vector of the arguments, we're printing it in the reversed order. 361 00:23:16,649 --> 00:23:21,179 And the magic line at the beginning is 362 00:23:21,179 --> 00:23:23,999 called a shebang and is the way that the 363 00:23:23,999 --> 00:23:26,159 shell will know how to run this program. 364 00:23:26,159 --> 00:23:30,509 You can always do something like 365 00:23:30,509 --> 00:23:34,379 "python script.py", and then "a b c" and that 366 00:23:34,379 --> 00:23:36,659 will work, always, like that. But 367 00:23:36,659 --> 00:23:39,119 what if we want to make this to be 368 00:23:39,119 --> 00:23:41,309 executable from the shell? The way the 369 00:23:41,309 --> 00:23:44,190 shell knows that it has to use python as the 370 00:23:44,190 --> 00:23:48,450 interpreter to run this file is using 371 00:23:48,450 --> 00:23:52,440 that first line. And that first line is 372 00:23:52,440 --> 00:23:56,620 giving it the path to where that thing lives. 373 00:23:58,500 --> 00:23:59,600 However, you might not know. 374 00:23:59,609 --> 00:24:01,830 Like, different machines will have probably 375 00:24:01,830 --> 00:24:04,049 different places where they put python 376 00:24:04,049 --> 00:24:06,090 and you might not want to assume where 377 00:24:06,090 --> 00:24:08,789 python is installed, or any other interpreter. 378 00:24:08,789 --> 00:24:16,379 So one thing that you can do is use the 379 00:24:16,380 --> 00:24:17,720 "env" command. 380 00:24:18,280 --> 00:24:21,560 You can also give arguments in the shebang, so 381 00:24:21,570 --> 00:24:23,940 what we're doing here is specifying 382 00:24:23,940 --> 00:24:29,720 run the "env" command, that is for pretty much every system, there are some exceptions, but like for 383 00:24:29,720 --> 00:24:31,550 pretty much every system it's is in 384 00:24:31,550 --> 00:24:33,620 "usr/bin", where a lot of binaries live, 385 00:24:33,620 --> 00:24:36,200 and then we're calling it with the 386 00:24:36,200 --> 00:24:38,570 argument "python". And then that will make 387 00:24:38,570 --> 00:24:42,020 use of the path environment variable 388 00:24:42,020 --> 00:24:43,580 that we saw in the first lecture. It's 389 00:24:43,580 --> 00:24:45,680 gonna search in that path for the Python 390 00:24:45,680 --> 00:24:48,620 binary and then it's gonna use that to 391 00:24:48,620 --> 00:24:50,480 interpret this file. And that will make 392 00:24:50,480 --> 00:24:52,490 this more portable so it can be run in 393 00:24:52,490 --> 00:24:57,520 my machine, and your machine and some other machine. 394 00:25:08,020 --> 00:25:12,140 Another thing is that the bash is not 395 00:25:12,140 --> 00:25:14,300 really like modern, it was 396 00:25:14,300 --> 00:25:16,340 developed a while ago. And sometimes 397 00:25:16,340 --> 00:25:18,890 it can be tricky to debug. By 398 00:25:18,890 --> 00:25:21,980 default, and the ways it will fail 399 00:25:21,980 --> 00:25:24,020 sometimes are intuitive like the way we 400 00:25:24,020 --> 00:25:26,180 saw before of like foo command not 401 00:25:26,180 --> 00:25:28,610 existing, sometimes it's not. So there's 402 00:25:28,610 --> 00:25:31,280 like a really nifty tool that we have 403 00:25:31,280 --> 00:25:34,310 linked in the lecture notes, which is called 404 00:25:34,310 --> 00:25:37,580 "shellcheck", that will kind of give you 405 00:25:37,580 --> 00:25:40,010 both warnings and syntactic errors 406 00:25:40,010 --> 00:25:43,250 and other things that you might not have quoted properly, 407 00:25:43,250 --> 00:25:46,040 or you might have misplaced spaces in 408 00:25:46,040 --> 00:25:50,060 your files. So for example for extremely simple "mcd.sh" 409 00:25:50,060 --> 00:25:51,980 file we're getting a couple 410 00:25:51,980 --> 00:25:54,800 of errors saying hey, surprisingly, 411 00:25:54,800 --> 00:25:56,090 we're missing a shebang, like this 412 00:25:56,090 --> 00:25:59,060 might not interpret it correctly if you're 413 00:25:59,060 --> 00:26:02,000 it at a different system. Also, this 414 00:26:02,000 --> 00:26:05,620 CD is taking a command and it might not 415 00:26:05,620 --> 00:26:08,960 expand properly so instead of using CD 416 00:26:08,960 --> 00:26:11,300 you might want to use something like CD 417 00:26:11,300 --> 00:26:14,540 and then an OR and then an "exit". We go 418 00:26:14,540 --> 00:26:16,490 back to what we explained earlier, what 419 00:26:16,490 --> 00:26:18,920 this will do is like if the 420 00:26:18,920 --> 00:26:21,860 CD doesn't end correctly, you cannot CD 421 00:26:21,860 --> 00:26:23,720 into the folder because either you 422 00:26:23,720 --> 00:26:25,250 don't have permissions, it doesn't exist... 423 00:26:25,250 --> 00:26:28,780 That will give a nonzero error 424 00:26:28,780 --> 00:26:32,420 command, so you will execute exit 425 00:26:32,420 --> 00:26:33,920 and that will stop the script 426 00:26:33,920 --> 00:26:35,810 instead of continue executing as if 427 00:26:35,810 --> 00:26:37,240 you were in a place that you are 428 00:26:37,240 --> 00:26:42,900 actually not in. And actually I haven't tested, but I 429 00:26:42,920 --> 00:26:47,179 think we can check for "example.sh" 430 00:26:47,179 --> 00:26:50,809 and here we're getting that we should be 431 00:26:50,809 --> 00:26:55,070 checking the exit code in a different way, because it's 432 00:26:55,070 --> 00:26:57,710 probably not the best way, doing it this 433 00:26:57,710 --> 00:27:01,580 way. One last remark I want to make 434 00:27:01,580 --> 00:27:05,090 is that when you're writing bash scripts 435 00:27:05,090 --> 00:27:07,159 or functions for that matter, 436 00:27:07,159 --> 00:27:09,080 there's kind of a difference between 437 00:27:09,080 --> 00:27:12,590 writing bash scripts in isolation like a 438 00:27:12,590 --> 00:27:14,149 thing that you're gonna run, and a thing 439 00:27:14,149 --> 00:27:16,100 that you're gonna load into your shell. 440 00:27:16,100 --> 00:27:19,850 We will see some of this in the command 441 00:27:19,850 --> 00:27:23,090 line environment lecture, where we will kind of 442 00:27:23,090 --> 00:27:29,059 be tooling with the bashrc and the sshrc. But in general, if you make 443 00:27:29,059 --> 00:27:31,370 changes to for example where you are, 444 00:27:31,370 --> 00:27:34,009 like if you CD into a bash script and you 445 00:27:34,009 --> 00:27:36,919 just execute that bash script, it won't CD 446 00:27:36,919 --> 00:27:39,980 into the shell are right now. But if you 447 00:27:39,980 --> 00:27:42,980 have loaded the code directly into 448 00:27:42,980 --> 00:27:45,559 your shell, for example you load... 449 00:27:45,559 --> 00:27:48,440 you source the function and then you execute 450 00:27:48,440 --> 00:27:50,269 the function then you will get those 451 00:27:50,269 --> 00:27:52,000 side effects. And the same goes for 452 00:27:52,000 --> 00:27:57,220 defining variables into the shell. 453 00:27:57,220 --> 00:28:03,950 Now I'm going to talk about some tools that I think are nifty when 454 00:28:03,950 --> 00:28:07,580 working with the shell. The first was 455 00:28:07,580 --> 00:28:09,799 also briefly introduced yesterday. 456 00:28:09,799 --> 00:28:13,309 How do you know what flags, or like 457 00:28:13,309 --> 00:28:15,320 what exact commands are. Like how I am 458 00:28:15,320 --> 00:28:21,889 supposed to know that LS minus L will list the files in a list format, or that 459 00:28:21,889 --> 00:28:25,789 if I do "move - i", it's gonna like prom me 460 00:28:25,789 --> 00:28:28,639 for stuff. For that what you have is the "man" 461 00:28:28,639 --> 00:28:30,730 command. And the man command will kind of 462 00:28:30,730 --> 00:28:33,590 have like a lot of information of how 463 00:28:33,590 --> 00:28:35,809 will you go about... so for example here it 464 00:28:35,809 --> 00:28:40,340 will explain for the "-i" flag, there are 465 00:28:40,340 --> 00:28:43,970 all these options you can do. That's 466 00:28:43,970 --> 00:28:45,620 actually pretty useful and it will work 467 00:28:45,620 --> 00:28:51,540 not only for really simple commands that come packaged with your OS 468 00:28:51,540 --> 00:28:55,809 but will also work with some tools that you install from the internet 469 00:28:55,809 --> 00:28:58,240 for example, if the person that did the 470 00:28:58,240 --> 00:29:01,390 installation made it so that the man 471 00:29:01,390 --> 00:29:03,399 package were also installed. So for example 472 00:29:03,399 --> 00:29:06,490 a tool that we're gonna cover in a bit 473 00:29:06,490 --> 00:29:12,370 which is called "ripgrep" and is called with RG, this didn't 474 00:29:12,370 --> 00:29:14,980 come with my system but it has installed 475 00:29:14,980 --> 00:29:17,230 its own man page and I have it here and 476 00:29:17,230 --> 00:29:21,700 I can access it. For some commands the 477 00:29:21,700 --> 00:29:25,029 man page is useful but sometimes it can be 478 00:29:25,029 --> 00:29:28,270 tricky to decipher because it's more 479 00:29:28,270 --> 00:29:30,399 kind of a documentation and a 480 00:29:30,399 --> 00:29:32,679 description of all the things the tool 481 00:29:32,679 --> 00:29:35,860 can do. Sometimes it will have 482 00:29:35,860 --> 00:29:37,720 examples but sometimes not, and sometimes 483 00:29:37,720 --> 00:29:41,620 the tool can do a lot of things so a 484 00:29:41,620 --> 00:29:45,250 couple of good tools that I use commonly 485 00:29:45,250 --> 00:29:50,289 are "convert" or "ffmpeg", which deal with images and video respectively and 486 00:29:50,289 --> 00:29:52,419 the man pages are like enormous. So there's 487 00:29:52,419 --> 00:29:54,850 one neat tool called "tldr" that 488 00:29:54,850 --> 00:29:58,240 you can install and you will have like 489 00:29:58,240 --> 00:30:02,710 some nice kind of explanatory examples 490 00:30:02,710 --> 00:30:05,470 of how you want to use this command. And you 491 00:30:05,470 --> 00:30:07,840 can always Google for this, but I find 492 00:30:07,840 --> 00:30:10,120 myself saving going into the 493 00:30:10,120 --> 00:30:12,640 browser, looking about some examples and 494 00:30:12,640 --> 00:30:14,919 coming back, whereas "tldr" are 495 00:30:14,919 --> 00:30:16,870 community contributed and 496 00:30:16,870 --> 00:30:19,210 they're fairly useful. Then, 497 00:30:19,210 --> 00:30:23,020 the one for "ffmpeg" has a lot of 498 00:30:23,020 --> 00:30:24,940 useful examples that are more nicely 499 00:30:24,940 --> 00:30:26,799 formatted (if you don't have a huge 500 00:30:26,799 --> 00:30:30,820 font size for recording). Or even 501 00:30:30,820 --> 00:30:33,250 simple commands like "tar", that have a lot 502 00:30:33,250 --> 00:30:35,470 of options that you are combining. So for 503 00:30:35,470 --> 00:30:37,840 example, here you can be combining 2, 3... 504 00:30:37,840 --> 00:30:41,710 different flags and it can not be 505 00:30:41,710 --> 00:30:43,419 obvious, when you want to combine 506 00:30:43,419 --> 00:30:48,429 different ones. That's how you 507 00:30:48,429 --> 00:30:54,850 would go about finding more about these tools. On the topic of finding, let's try 508 00:30:54,850 --> 00:30:58,690 learning how to find files. You can 509 00:30:58,690 --> 00:31:03,100 always go "ls", and like you can go like 510 00:31:03,100 --> 00:31:05,950 "ls project1", and 511 00:31:05,950 --> 00:31:08,559 keep LS'ing all the way through. But 512 00:31:08,559 --> 00:31:11,740 maybe, if we already know that we want 513 00:31:11,740 --> 00:31:15,450 to look for all the folders called 514 00:31:15,450 --> 00:31:19,000 "src", then there's probably a better command 515 00:31:19,000 --> 00:31:21,400 for doing that. And that's "find". 516 00:31:21,460 --> 00:31:26,679 Find is the tool that, pretty much comes with every UNIX system. And find, 517 00:31:26,679 --> 00:31:35,230 we're gonna give it... here we're saying we want to call find in the 518 00:31:35,230 --> 00:31:37,510 current folder, remember that "." stands 519 00:31:37,510 --> 00:31:40,149 for the current folder, and we want the 520 00:31:40,149 --> 00:31:46,539 name to be "src" and we want the type to be a directory. And by typing that it's 521 00:31:46,539 --> 00:31:49,870 gonna recursively go through the current 522 00:31:49,870 --> 00:31:52,330 directory and look for all these files, 523 00:31:52,330 --> 00:31:58,659 or folders in this case, that match this pattern. Find has a lot of useful 524 00:31:58,659 --> 00:32:01,840 flags. So for example, you can even test 525 00:32:01,840 --> 00:32:05,440 for the path to be in a way. Here we're 526 00:32:05,440 --> 00:32:08,230 saying we want some number of folders, 527 00:32:08,230 --> 00:32:09,909 we don't really care how many folders, 528 00:32:09,909 --> 00:32:13,179 and then we care about all the Python 529 00:32:13,179 --> 00:32:17,830 scripts, all the things with the extension ".py", that are within a 530 00:32:17,830 --> 00:32:19,899 test folder. And we're also checking, just in 531 00:32:19,899 --> 00:32:21,519 cases really but we're checking just 532 00:32:21,519 --> 00:32:24,460 that it's also a type F, which stands for 533 00:32:24,460 --> 00:32:28,710 file. We're getting all these files. 534 00:32:28,710 --> 00:32:32,169 You can also use different flags for things 535 00:32:32,169 --> 00:32:34,000 that are not the path or the name. 536 00:32:34,000 --> 00:32:38,160 You could check things that have been 537 00:32:38,160 --> 00:32:42,060 modified ("-mtime" is for the modification time), things that have been 538 00:32:42,070 --> 00:32:44,540 modified in the last day, which is gonna 539 00:32:44,559 --> 00:32:46,659 be pretty much everything. So this is gonna print 540 00:32:46,659 --> 00:32:49,029 a lot of the files we created and files 541 00:32:49,029 --> 00:32:51,850 that were already there. You can even 542 00:32:51,850 --> 00:32:54,960 use other things like size, the owner, 543 00:32:54,960 --> 00:32:59,080 permissions, you name it. What is even more 544 00:32:59,080 --> 00:33:01,870 powerful is, "find" can find stuff 545 00:33:01,870 --> 00:33:04,269 but it also can do stuff when you 546 00:33:04,269 --> 00:33:10,690 find those files. So we could look for all 547 00:33:10,690 --> 00:33:14,080 the files that have a TMP 548 00:33:14,080 --> 00:33:18,160 extension, which is a temporary extension, and 549 00:33:18,160 --> 00:33:22,720 then, we can tell "find" that for every one of those files, 550 00:33:22,720 --> 00:33:26,350 just execute the "rm" command for them. And 551 00:33:26,350 --> 00:33:29,050 that will just be calling "rm" with all 552 00:33:29,050 --> 00:33:32,350 these files. So let's first execute it 553 00:33:32,350 --> 00:33:35,760 without, and then we execute it with it. 554 00:33:35,760 --> 00:33:38,950 Again, as with the command line 555 00:33:38,950 --> 00:33:41,470 philosophy, it looks like nothing 556 00:33:41,470 --> 00:33:48,070 happened. But since we have a zero error code, something 557 00:33:48,070 --> 00:33:49,540 happened - just that everything went 558 00:33:49,540 --> 00:33:51,490 correct and everything is fine. And now, 559 00:33:51,490 --> 00:33:57,810 if we look for these files, they aren't there anymore. 560 00:33:57,810 --> 00:34:02,950 Another nice thing about the shell in general is that there are 561 00:34:02,950 --> 00:34:05,890 these tools, but people will keep 562 00:34:05,890 --> 00:34:08,230 finding new ways, so alternative 563 00:34:08,230 --> 00:34:12,220 ways of writing these tools. It's nice to know about it. So, for 564 00:34:12,220 --> 00:34:20,020 example find if you just want to match the things that end in "tmp" 565 00:34:20,020 --> 00:34:24,190 it can be sometimes weird to do this thing, it has a long command. 566 00:34:24,190 --> 00:34:27,760 There's things like "fd", 567 00:34:27,760 --> 00:34:32,320 for example, that is a shorter command that by default will use regex 568 00:34:32,320 --> 00:34:34,899 and will ignore your gitfiles, so you 569 00:34:34,899 --> 00:34:38,020 don't even search for them. It 570 00:34:38,020 --> 00:34:42,879 will color-code, it will have better Unicode support... It's nice to 571 00:34:42,879 --> 00:34:45,040 know about some of these tools. But, again, 572 00:34:45,040 --> 00:34:52,149 the main idea is that if you are aware that these tools exist, you can 573 00:34:52,149 --> 00:34:53,740 save yourself a lot of time from doing 574 00:34:53,740 --> 00:34:57,660 kind of menial and repetitive tasks. 575 00:34:57,660 --> 00:35:00,010 Another command to bear in mind is like 576 00:35:00,010 --> 00:35:01,990 "find". Some of you may be 577 00:35:01,990 --> 00:35:04,300 wondering, "find" is probably just 578 00:35:04,300 --> 00:35:06,520 actually going through a directory 579 00:35:06,520 --> 00:35:09,580 structure and looking for things but 580 00:35:09,580 --> 00:35:11,260 what if I'm doing a lot of "finds" a day? 581 00:35:11,260 --> 00:35:12,850 Wouldn't it be better, doing kind of 582 00:35:12,850 --> 00:35:18,790 a database approach and build an index first, and then use that index 583 00:35:18,790 --> 00:35:21,520 and update it in some way. Well, actually 584 00:35:21,520 --> 00:35:23,380 most Unix systems already do it and 585 00:35:23,380 --> 00:35:28,170 this is through the "locate" command and 586 00:35:28,170 --> 00:35:31,690 the way that the locate will 587 00:35:31,690 --> 00:35:35,470 be used... it will just look for paths in 588 00:35:35,470 --> 00:35:38,680 your file system that have the substring 589 00:35:38,680 --> 00:35:44,710 that you want. I actually don't know if it will work... Okay, it worked. Let me try to 590 00:35:44,710 --> 00:35:49,840 do something like "missing-semester". 591 00:35:51,840 --> 00:35:53,950 You're gonna take a while but 592 00:35:53,950 --> 00:35:56,109 it found all these files that are somewhere 593 00:35:56,109 --> 00:35:57,730 in my file system and since it has 594 00:35:57,730 --> 00:36:01,750 built an index already on them, it's much 595 00:36:01,750 --> 00:36:05,680 faster. And then, to keep it updated, 596 00:36:05,680 --> 00:36:11,980 using the "updatedb" command that is running through cron, 597 00:36:13,840 --> 00:36:18,490 to update this database. Finding files, again, is 598 00:36:18,490 --> 00:36:23,230 really useful. Sometimes you're actually concerned about, not the files themselves, 599 00:36:23,230 --> 00:36:26,740 but the content of the files. For that 600 00:36:26,740 --> 00:36:31,420 you can use the grep command that we 601 00:36:31,420 --> 00:36:33,880 have seen so far. So you could do 602 00:36:33,880 --> 00:36:37,740 something like grep foobar in MCD, it's there. 603 00:36:37,740 --> 00:36:43,690 What if you want to, again, recursively search through the current 604 00:36:43,690 --> 00:36:45,760 structure and look for more files, right? 605 00:36:45,760 --> 00:36:48,700 We don't want to do this manually. 606 00:36:48,700 --> 00:36:51,220 We could use "find", and the "-exec", but 607 00:36:51,220 --> 00:36:58,920 actually "grep" has the "-R" flag that will go through the entire 608 00:36:58,920 --> 00:37:03,609 directory, here. And it's telling us 609 00:37:03,609 --> 00:37:06,579 that oh we have the foobar line in example.sh 610 00:37:06,579 --> 00:37:09,279 at these three places and in 611 00:37:09,279 --> 00:37:14,589 this other two places in foobar. This can be 612 00:37:14,589 --> 00:37:16,900 really convenient. Mainly, the 613 00:37:16,900 --> 00:37:18,940 use case for this is you know you have 614 00:37:18,940 --> 00:37:21,910 written some code in some programming 615 00:37:21,910 --> 00:37:23,859 language, and you know it's somewhere in 616 00:37:23,859 --> 00:37:26,200 your file system but you actually don't 617 00:37:26,200 --> 00:37:28,599 know. But you can actually quickly search. 618 00:37:28,600 --> 00:37:32,980 So for example, I can quickly search 619 00:37:35,660 --> 00:37:40,320 for all the Python files that I have in my 620 00:37:40,329 --> 00:37:45,460 scratch folder where I used the request library. 621 00:37:45,460 --> 00:37:47,589 And if I run this, it's giving me 622 00:37:47,589 --> 00:37:50,890 through all these files, exactly in 623 00:37:50,890 --> 00:37:53,650 what line it has been found. And here 624 00:37:53,650 --> 00:37:56,260 instead of using grep, which is fine, 625 00:37:56,260 --> 00:37:58,930 you could also do this, I'm using "ripgrep", 626 00:37:58,930 --> 00:38:05,260 which is kind of the same idea but again trying to bring some more 627 00:38:05,260 --> 00:38:09,730 niceties like color coding or file 628 00:38:09,730 --> 00:38:16,480 processing and other things. It think it has, also, unicode support. It's also pretty 629 00:38:16,480 --> 00:38:22,829 fast so you are not paying like a trade-off on this being slower and 630 00:38:22,829 --> 00:38:25,420 there's a lot of useful flags. You 631 00:38:25,420 --> 00:38:27,670 can say, oh, I actually want to get some 632 00:38:27,670 --> 00:38:30,460 context around those results. 633 00:38:33,040 --> 00:38:36,400 So I want to get like five lines of context around 634 00:38:36,400 --> 00:38:42,819 that, so you can see where that import lives and see code around it. 635 00:38:42,819 --> 00:38:44,170 Here in the import it's not really useful 636 00:38:44,170 --> 00:38:45,819 but like if you're looking for where you 637 00:38:45,819 --> 00:38:49,720 use the function, for example, it will 638 00:38:49,720 --> 00:38:54,010 be very handy. We can also do things like 639 00:38:54,010 --> 00:38:59,170 we can search, for example here,. 640 00:38:59,170 --> 00:39:04,839 A more advanced use, we can say, 641 00:39:04,840 --> 00:39:11,580 "-u" is for don't ignore hidden files, sometimes 642 00:39:12,520 --> 00:39:16,359 you want to be ignoring hidden files, except if you want to 643 00:39:16,359 --> 00:39:23,500 search config files, that are by default hidden. Then, instead of printing 644 00:39:23,500 --> 00:39:28,400 the matches, we're asking to do something that would be kind of hard, I think, 645 00:39:28,400 --> 00:39:31,380 to do with grep, out of my head, which is 646 00:39:31,390 --> 00:39:34,569 "I want you to print all the files that 647 00:39:34,569 --> 00:39:37,750 don't match the pattern I'm giving you", which 648 00:39:37,750 --> 00:39:40,030 may be a weird thing to ask here but 649 00:39:40,030 --> 00:39:42,940 then we keep going... And this pattern here 650 00:39:42,940 --> 00:39:45,790 is a small regex which is saying 651 00:39:45,790 --> 00:39:48,099 at the beginning of the line I have a 652 00:39:48,099 --> 00:39:51,190 "#" and a "!", and that's a shebang. 653 00:39:51,190 --> 00:39:53,470 Like that, we're searching here for all 654 00:39:53,470 --> 00:39:56,650 the files that don't have a shebang 655 00:39:56,650 --> 00:39:59,369 and then we're giving it, here, 656 00:39:59,369 --> 00:40:02,470 a "-t sh" to only look for "sh" 657 00:40:02,470 --> 00:40:07,660 files, because maybe all your Python or text files are fine 658 00:40:07,660 --> 00:40:10,000 without a shebang. And here it's telling us 659 00:40:10,000 --> 00:40:13,020 "oh, MCD is obviously missing a shebang" 660 00:40:14,760 --> 00:40:16,660 We can even... It has like some 661 00:40:16,660 --> 00:40:19,119 nice flags, so for example if we 662 00:40:19,120 --> 00:40:21,360 include the "stats" flag 663 00:40:28,700 --> 00:40:34,119 it will get all these results but it will also tell us information about all 664 00:40:34,119 --> 00:40:35,410 the things that it searched. For example, 665 00:40:35,410 --> 00:40:40,390 the number of matches that it found, the lines, the file searched, 666 00:40:40,390 --> 00:40:44,040 the bytes that it printed, &c. 667 00:40:44,040 --> 00:40:47,160 Similar as with "fd", sometimes it's not as useful 668 00:40:48,400 --> 00:40:50,619 using one specific tool or another and 669 00:40:50,620 --> 00:40:55,780 in fact, as ripgrep, there are several other tools. Like "ack", 670 00:40:55,780 --> 00:40:57,700 is the original grep alternative that was 671 00:40:57,700 --> 00:41:00,670 written. Then the silver searcher, 672 00:41:00,670 --> 00:41:04,089 "ag", was another one... and they're all 673 00:41:04,089 --> 00:41:05,589 pretty much interchangeable so 674 00:41:05,589 --> 00:41:07,630 maybe you're at a system that has one and 675 00:41:07,630 --> 00:41:09,670 not the other, just knowing that you can 676 00:41:09,670 --> 00:41:12,040 use these things with these tools can be 677 00:41:12,040 --> 00:41:15,549 fairly useful. Lastly, I want to cover 678 00:41:15,549 --> 00:41:19,780 how you go about, not finding files or code, but how you go about 679 00:41:19,780 --> 00:41:22,540 finding commands that you already 680 00:41:22,540 --> 00:41:30,160 some time figured out. The first, obvious way is just using the up arrow, 681 00:41:30,160 --> 00:41:34,540 and slowly going through all your history, looking for these matches. 682 00:41:34,540 --> 00:41:36,490 This is actually not very efficient, as 683 00:41:36,490 --> 00:41:42,579 you probably guessed. So the bash has ways to do this more easily. 684 00:41:42,579 --> 00:41:44,619 There is the "history" command, that will 685 00:41:44,619 --> 00:41:49,180 print your history. Here I'm in zsh and it only prints some of my history, but 686 00:41:49,180 --> 00:41:54,069 if I say, I want you to print everything from the beginning of time, it will print 687 00:41:54,069 --> 00:41:58,220 everything from the beginning of whatever this history is. 688 00:41:58,220 --> 00:42:00,700 And since this is a lot of results, 689 00:42:00,700 --> 00:42:02,589 maybe we care about the ones where we 690 00:42:02,589 --> 00:42:08,490 use the "convert" command to go from some type of file to some other type of file. 691 00:42:08,490 --> 00:42:12,940 Some image, sorry. Then, we're getting all 692 00:42:12,940 --> 00:42:15,849 these results here, about all the ones 693 00:42:15,849 --> 00:42:18,120 that match this substring. 694 00:42:21,280 --> 00:42:24,609 Even more, pretty much all shells by default will 695 00:42:24,609 --> 00:42:27,130 link "Ctrl+R", the keybinding, 696 00:42:27,130 --> 00:42:29,680 to do backward search. Here we 697 00:42:29,680 --> 00:42:31,569 have backward search, where we can 698 00:42:31,569 --> 00:42:34,750 type "convert" and it's finding the 699 00:42:34,750 --> 00:42:36,609 command that we just typed. And if we just 700 00:42:36,609 --> 00:42:38,619 keep hitting "Ctrl+R", it will 701 00:42:38,619 --> 00:42:41,740 kind of go through these matches and 702 00:42:41,740 --> 00:42:44,260 it will let re-execute it 703 00:42:44,260 --> 00:42:49,240 in place. Another thing that you can do, 704 00:42:49,240 --> 00:42:51,069 related to that, is you can use this 705 00:42:51,069 --> 00:42:53,829 really nifty tool called "fzf", which is 706 00:42:53,829 --> 00:42:56,280 like a fuzzy finder, like it will... 707 00:42:57,100 --> 00:42:58,480 It will let you do kind of 708 00:42:58,480 --> 00:43:02,200 like an interactive grep. We could do 709 00:43:02,200 --> 00:43:06,369 for example this, where we can cat our 710 00:43:06,369 --> 00:43:10,030 example.sh command, that will print 711 00:43:10,030 --> 00:43:11,680 print to the standard output, and then we 712 00:43:11,680 --> 00:43:14,290 can pipe it through fzf. It's just getting 713 00:43:14,290 --> 00:43:18,490 all the lines and then we can interactively look for the 714 00:43:18,490 --> 00:43:21,849 string that we care about. And the nice 715 00:43:21,849 --> 00:43:26,349 thing about fzf is that, if you enable the default bindings, it will bind to 716 00:43:26,349 --> 00:43:33,670 your "Ctrl+R" shell execution and now 717 00:43:33,670 --> 00:43:36,490 you can quickly and dynamically like 718 00:43:36,490 --> 00:43:41,700 look for all the times you try to convert a favicon in your history. 719 00:43:42,020 --> 00:43:46,375 And it's also like fuzzy matching, whereas like by default in grep 720 00:43:46,375 --> 00:43:49,420 or these things you have to write a regex or some 721 00:43:49,420 --> 00:43:52,360 expression that will match within here. 722 00:43:52,360 --> 00:43:54,609 Here I'm just typing "convert" and "favicon" and 723 00:43:54,609 --> 00:43:57,369 it's just trying to do the best scan, 724 00:43:57,369 --> 00:44:01,349 doing the match in the lines it has. 725 00:44:01,349 --> 00:44:06,190 Lastly, a tool that probably you have already seen, that I've been using 726 00:44:06,190 --> 00:44:08,410 for not retyping these extremely long 727 00:44:08,410 --> 00:44:13,080 commands is this "history substring search", where 728 00:44:13,940 --> 00:44:15,660 as I type in my shell, 729 00:44:15,670 --> 00:44:19,630 and both F fail to mention but both face 730 00:44:19,630 --> 00:44:22,760 which I think was originally introduced, this concept, and then 731 00:44:22,760 --> 00:44:25,760 zsh has a really nice implementation) 732 00:44:25,760 --> 00:44:26,800 what it'll let you do is 733 00:44:26,800 --> 00:44:31,300 as you type the command, it will dynamically search back in your 734 00:44:31,300 --> 00:44:34,420 history to the same command that has a common prefix, 735 00:44:34,980 --> 00:44:36,900 and then, if you... 736 00:44:39,100 --> 00:44:42,100 it will change as the match list stops 737 00:44:42,100 --> 00:44:44,110 working and then as you do the 738 00:44:44,120 --> 00:44:49,760 right arrow you can select that command and then re-execute it. 739 00:45:05,800 --> 00:45:09,920 We've seen a bunch of stuff... I think I have 740 00:45:09,940 --> 00:45:16,180 a few minutes left so I'm going to cover a couple of tools to do 741 00:45:16,180 --> 00:45:20,060 really quick directory listing and directory navigation. 742 00:45:20,060 --> 00:45:30,020 So you can always use the "-R" to recursively list some directory structure, 743 00:45:30,020 --> 00:45:35,160 but that can be suboptimal, I cannot really make sense of this easily. 744 00:45:36,340 --> 00:45:44,460 There's tool called "tree" that will be the much more friendly form of 745 00:45:44,460 --> 00:45:47,500 printing all the stuff, it will also color code based on... 746 00:45:47,500 --> 00:45:50,680 here for example "foo" is blue because it's a directory and 747 00:45:50,680 --> 00:45:55,100 this is red because it has execute permissions. 748 00:45:55,100 --> 00:46:00,220 But we can go even further than that. There's really nice tools 749 00:46:00,220 --> 00:46:04,580 like a recent one called "broot" that will do the same thing but here 750 00:46:04,580 --> 00:46:07,300 for example instead of doing this thing of listing 751 00:46:07,300 --> 00:46:09,160 every single file, for example in bar 752 00:46:09,160 --> 00:46:11,400 we have these "a" through "j" files, 753 00:46:11,400 --> 00:46:14,260 it will say "oh there are more, unlisted here". 754 00:46:15,080 --> 00:46:18,200 I can actually start typing and it will again 755 00:46:18,200 --> 00:46:21,540 again facily match to the files that are there 756 00:46:21,540 --> 00:46:24,800 and I can quickly select them and navigate through them. 757 00:46:24,800 --> 00:46:28,380 So, again, it's good to know that 758 00:46:28,380 --> 00:46:33,340 these things exist so you don't lose a large amount of time 759 00:46:34,240 --> 00:46:36,180 going for these files. 760 00:46:37,880 --> 00:46:40,500 There are also, I think I have it installed 761 00:46:40,500 --> 00:46:44,829 also something more similar to what you would expect your OS to have, 762 00:46:44,829 --> 00:46:49,960 like Nautilus or one of the Mac finders that have like an 763 00:46:49,960 --> 00:46:59,260 interactive input where you can just use your navigation arrows and quickly explore. 764 00:46:59,260 --> 00:47:03,849 It might be overkill but you'll be surprised how quickly you can 765 00:47:03,849 --> 00:47:07,839 make sense of some directory structure by just navigating through it. 766 00:47:07,840 --> 00:47:12,780 And pretty much all of these tools will let you edit, copy files... 767 00:47:12,780 --> 00:47:16,880 if you just look for the options for them. 768 00:47:17,600 --> 00:47:20,100 The last addendum is kind of going places. 769 00:47:20,100 --> 00:47:24,480 We have "cd", and "cd" is nice, it will get you 770 00:47:26,120 --> 00:47:30,060 to a lot of places. But it's pretty handy if 771 00:47:30,069 --> 00:47:33,190 you can like quickly go places, 772 00:47:33,190 --> 00:47:36,730 either you have been to recently or that 773 00:47:36,730 --> 00:47:40,599 you go frequently. And you can do this in 774 00:47:40,599 --> 00:47:42,520 many ways there's probably... you can start 775 00:47:42,520 --> 00:47:44,319 thinking, oh I can make bookmarks, I can 776 00:47:44,319 --> 00:47:46,660 make... I can make aliases in the shell, 777 00:47:46,660 --> 00:47:49,020 that we will cover at some point, 778 00:47:49,020 --> 00:47:53,020 symlinks... But at this point, 779 00:47:53,020 --> 00:47:54,910 programmers have like built all these 780 00:47:54,910 --> 00:47:56,799 tools, so programmers have already figured 781 00:47:56,799 --> 00:47:59,520 out a really nice way of doing this. 782 00:47:59,520 --> 00:48:01,930 One way of doing this is using what is 783 00:48:01,930 --> 00:48:05,760 called "auto jump", which I think is not loaded here... 784 00:48:14,140 --> 00:48:20,100 Okay, don't worry. I will cover it in the command line environment. 785 00:48:21,960 --> 00:48:25,579 I think it's because I disabled the "Ctrl+R" and that also 786 00:48:25,579 --> 00:48:31,309 affected other parts of the script. I think at this point if anyone has 787 00:48:31,309 --> 00:48:35,480 any questions that are related to this, I'll be more than happy to answer 788 00:48:35,480 --> 00:48:37,509 them, if anything was left unclear. 789 00:48:37,509 --> 00:48:42,859 Otherwise, a there's a bunch of exercises that we wrote, kind of 790 00:48:42,859 --> 00:48:46,549 touching on these topics and we encourage you to try them and 791 00:48:46,549 --> 00:48:48,559 come to office hours, where we can help 792 00:48:48,559 --> 00:48:54,569 you figure out how to do them, or some bash quirks that are not clear.