Lecture 11: Q&A (2020)

I guess we should do an intro to to this as well,

so this is a just sort of a

free-form Q&A lecture where you, as in
the two people sitting here, but also

everyone at home who did not come here
in person get to ask questions and we

have a bunch of questions people asked
in advance but you can also ask

additional questions during, for the two
of you who are here, you can do it either

by raising your hand or you can submit it on
the forum and be anonymous, it's up to you

regardless though, what we're gonna
do is just go through some of the

questions have been asked and try to
give as helpful answers as we can

although they are unprepared on our side and

yeah that's the plan I guess we go

from popular to least popular

fire away

all right so for our first question any

recommendations on learning operating
system related topics like processes,

virtual memory, interrupts,
memory management, etc

so I think this is a

is an interesting question because these
are really low level concepts that often

do not matter, unless you have to
deal with this in some capacity,

right so

one instance where this matters is you're
writing really low level code like

you're implementing a kernel or something
like that, or you want to

just hack on the Linux kernel.

It's rare otherwise that you need to work with

especially like virtual memory and
interrupts and stuff yourself

processes, I think are a more general concept
that we've talked a little bit about in

this class as well and tools like
htop, pgrep, kill, and signals and

that sort of stuff

in terms of learning it

maybe one of the best ways, is to try to
take either an introductory class on the

topic, so for example MIT has a class
called 6.828, which is where

you essentially build and develop your
own operating system based on some code

that you're given, and all of those labs
are publicly available and all the

resources for the class are publicly available,
and so that is a good way to

really learn them is by doing them yourself.

There are also various

tutorials online that basically guide
you through how do you write a kernel

from scratch. Not necessarily a very
elaborate one, not one you would want

to run any real software on, but just to
teach you the basics and so that would

be another thing to look up.

Like how do I write a kernel in and then your

language of choice. You will probably not
find one that lets you do it in Python

but in like C, C++, Rust, there
are a bunch of topics like this

one other note on operating systems

so like Jon mentioned MIT has a 6.828 class but

if you're looking for a more high-level
overview, not necessarily programming or

an operating system, but just learning about
the concepts another good resource

is a book called "Modern Operating
Systems" by Andy Tannenbaum

there's also actually a book called the "The FreeBSD
Operating System" which is really good,

It doesn't go through Linux, but it goes
through FreeBSD and the BSD kernel is

arguably better organized than the Linux
one and better documented and so it

might be a gentler introduction to some of those
topics than trying to understand Linux

You want to check it as answered?

- Yes + Nice

Answered

For our next question,

What are some of the tools you'd
prioritize learning first?

- Maybe we can all go through and
give our opinion on this? + Yeah

Tools to prioritize learning first?

I think learning your editor well,
just serves you in all capacities

like being efficient at editing files,
is just like a majority of

what you're going to spend your time doing.
And in general, just using your

keyboard more and your mouse less. It means
that you get to spend more of your

time doing useful things and
less of your time moving

I think that would be my top priority,

so I would say that for what

tool to prioritize will depend
on what exactly you're doing

I think the core idea is you should try
to find the types of tasks that you are

doing repetitively and so

if you are doing some sort of like
machine learning workload and

you find yourself using Jupyter notebooks,
like the one we presented

yesterday, a lot. Then again, using
a mouse for that might not be

the best idea and you want to familiarize
with the keyboard shortcuts

and pretty much with anything you will
end up figuring out that there are some

repetitive tasks, and you're running a
computer, and just trying to figure out

oh there's probably a better way to do this

be it a terminal, be it an editor

And it might be really interesting to
learn to use some of the topics that

we have covered, but if they're not
extremely useful in a everyday

basis then it might not be worth prioritizing them

Out of the topics

covered in this class, in my opinion, two
of the most useful things are version

control and text editors, and I think they're
a little bit different from each

other, in the sense that text editors I
think are really useful to learn well,

but it was probably the case that before
we started using Vim and all its fancy

keyboard shortcuts you had some other
text editor you were using before and

you could edit text just fine maybe a little
bit inefficiently, whereas I think

version control is another really useful
skill and that's one where if you don't

really know the tool properly, it can actually
lead to some problems like loss

of data or just inability to collaborate
properly with people. So I

think version control is one of the first
things that's worth learning well.

Yeah, I agree with that. I think
learning a tool like Git is just

gonna save you so much heartache down the line.

It, also, to add on to that,

it really helps you collaborate with others,
and Anish touched a little bit on GitHub

in the last lecture, and just learning
to use that tool well in order

to work on larger software projects
that other people are working on is

an invaluable skill.

For our next question,

"When do I use Python versus a

Bash script versus some other language?"

This is tough, because I think this comes

down to what Jose was saying earlier too,

that it really depends on
what you're trying to do.

For me, I think for Bash scripts in particular,

Bash scripts are for

automating running a bunch of commands.
You don't want to write any

other, like, business logic in Bash.

Like, it is just for, 'I want to run these

commands, in this order... maybe with
arguments?' But - but, like, even that,

it's unclear that you want a Bash script
once you start taking arguments.

Similarly, like, once you start doing any
kind of, like, text processing, or

configuration, all that,

reach for a language that is... a more, a more serious

programming language than Bash is.

Bash is really for short, one-off

scripts, or ones that have a very well-defined
use case, on the terminal, in

the shell, probably.

For a slightly more concrete guideline,
you might say, 'Write a

Bash script if it's less than a hundred
lines of code or so', but once it gets

beyond that point, Bash is kind of
unwieldy, and it's probably worth

switching to a more serious programming
language, like Python.

And, to add to that,

I would say, like, I found myself writing,
sometimes, scripts in Python, because

if I have already solved some subproblem
that covers part of the problem in Python,

I find it much easier to compose the
previous solution that I found out in

Python than just try to reuse Bash code,
that I don't find as reusable as Python.

And in the same way it's kind of nice that
a lot of people have written something

like Python libraries or like Ruby libraries
to do a lot of these things,

whereas, in Bash, it's kind of hard
to have, like, code reuse.

And, in fact,

I think to add to that, usually, if you
find a library, in some language that

helps with the task you're trying to
do, use that language for the job.

And in Bash, there are no libraries. There
are only the programs on your computer.

So you probably don't want to use
it, unless like there's a program

you can just invoke. I do think another
thing worth remembering about Bash is:

Bash is really hard to get right.

It's very easy to get it right for the particular
use case you're trying to solve right now,

but, like, things like,

"What if one of the filenames has a space in it?"

It has caused so many bugs, and so

many problems in Bash scripts. And, if you
use a - a real programming language, then

those problems just go away.

Yes! Checked it.

For our next question,

what is the difference between sourcing
a script, and executing that script?

Ooh. So, this, actually, we got in office
hours a - a while back, as well, which is,

'Aren't they the same? Like, aren't they
both just running the Bash script?'

And, it is true

both of these will end up executing the
lines of code that are in the script.

The ways in which they differ is that
sourcing a script is telling your

current Bash script, your current Bash
session, to execute that program,

whereas the other one is, 'Start up a new instance
of Bash, and run the program there, instead.'

And, this matters for things like... Imagine that
"script.sh" tries to change directories.

If you are running the script,
as in the second invocation,

"./script.sh", then the new
process is going to change

directories. But, by the time that script
exits, and returns to your shell,

your shell still remains in the same place. However,
if you do "cd" in a script, and you "source" it,

your current instance of Bash is the
one that ends up running it, and

so, it ends up "cd"-ing where you are.

This is also why, if you define functions,

for example, that you may want to
execute in your shell session,

you need to source the script, not run it,

because if you run it, that function
will be defined in the

instance of Bash,

in the Bash process that gets launched, but it
will not be defined in your current shell.

I think those are two of the biggest
differences between the two.

Next question...

"What are the places where various packages and tools
are stored and how does referencing them work?

What even is /bin or /lib?"

So, as we covered in the first lecture,
there is this PATH environment variable,

which is like a semicolon-separated-
string of all the places

where your shell is gonna look for binaries.
And, if you just do something like

"echo $PATH", you're gonna get this list;

all these places are gonna
be consulted, in order.

It's gonna go through all of them, and, in fact,

- There is already... Did we cover which? + Yeah

So, if you run "which", and a specific command,

the shell is actually gonna tell
you where it's finding this (command).

Beyond that,

there is like some conventions where a lot
of programs will install their binaries

and they're like /usr/bin (or at
least they will include symlinks)

in /usr/bin so you can find them

There's also a /usr/local/bin

There are special directories. For example,
/usr/sbin it's only for sudo user and

some of these conventions are slightly
different between different distros so

I know like some distros for example install
the user libraries under /opt for example

Yeah I think one thing just
to talk a little bit of more

about /bin and then Anish maybe you can
do the other folders so when it comes to

/bin the convention

There are conventions, and the conventions are
usually /bin are for essential system utilities

/usr/bin are for user programs and

/usr/local/bin are for user
compiled programs, sort of

so things that you installed that you intend
the user to run, are in /usr/bin

things that a user has compiled themselves and stuck
on your system, probably goes in /usr/local/bin

but again, this varies a lot from machine
to machine, and distro to distro

On Arch Linux, for example, /bin
is a symlink to /usr/bin

They're the same and as Jose mentioned, there's
also /sbin which is for programs that are

intended to only be run as root, that
also varies from distro to distro

whether you even have that directory, and
on many systems like /usr/local/bin

might not even be in your PATH, or
might not even exist on your system

On BSD on the other hand /usr/local/bin
is often used a lot more heavily

yeah so

What we were talking about so far, these
are all ways that files and folders are

organized on Linux things or Linux or
BSD things vary a little bit between

that and macOS or other platforms

I think for the specific locations,

if you to know exactly what it's
used for, you can look it up

But some general patterns to keep in mind or anything
with /bin in it has binary executable programs in it,

anything with \lib in it, has
libraries in it so things that

programs can link against, and then some
other things that are useful to know are

there's a /etc on many systems, which
has configuration files in it and

then there's /home, which underneath that directory
contains each user's home directory

so like on a linux box my username
or if it's Anish will

correspond to a home directory /home/anish

Yeah I guess there are

a couple of others like /tmp is usually
a temporary directory that gets

erased when you reboot not always but sometimes,
you should check on your system

There's a /var which often holds like
files the change over time so

these these are usually going to be things
like lock files for package managers

they're gonna be things like log files
files to keep track of process IDs

then there's /dev which shows devices so

usually so these are special files that
correspond to devices on your system we

talked about /sys, Anish mentioned /etc

/opt is a common one for just like third-party
software that basically it's usually for

companies ported their software to Linux
but they don't actually understand what

running software on Linux is like, and
so they just have a directory with all

their stuff in it and when those get installed
they usually get installed into /opt

I think those are the ones off the top of my head

yeah

And we will list these in our lecture notes
which will produce after this lecture

Next question

Should I apt-get install a Python whatever

package or pip install that package

so this is a good question that I think at

a higher level this question is asking
should I use my systems package manager

to install things or should I use some other
package manager. Like in this case

one that's more specific to a particular
language. And the answer here is also

kind of it depends, sometimes it's nice
to manage things using a system package

manager so everything can be installed
and upgraded in a single place but

I think oftentimes whatever is available
in the system repositories the things

you can get via a tool like
apt-get or something similar

might be slightly out of date compared to
the more language specific repository

so for example a lot of the Python packages
I use I really want the most

up-to-date version and so
I use pip to install them

Then, to extend on that is

sometimes the case the system packages
might require some other

dependencies that you might not have realized
about, and it's also might be

the case or like for some systems,
at least for like alpine Linux they

don't have wheels for like a lot of the
Python packages so it will just take

longer to compile them, it will take more
space because they have to compile them

from scratch. Whereas if you just go
to pip, pip has binaries for a lot of

different platforms and that will probably work

You also should be aware that pip might not do
the exact same thing in different computers

So, for example, if you are in a kind of laptop
or like a desktop that is running like

a x86 or x86_64 you probably have binaries,
but if you're running something

like Raspberry Pi or some other kind of
embedded device. These are running on a

different kind of hardware architecture
and you might not have binaries

I think that's also good to take into account,
in that case in might be worthwhile to

use the system packages just because they
will take much shorter to get them

than to just to compile from scratch
the entire Python installation

Apart from that, I don't think I can think of any exceptions
where I would actually use the system packages

instead of the Python provided ones

So, one other thing to keep in mind is that

sometimes you will have more than one
program on your computer and you might

be developing more than one program on
your computer and for some reason not

all programs are always built with the latest
version of things, sometimes they

are a little bit behind, and when you
install something system-wide you can

only... depends on your exact system,
but often you just have one version

what pip lets you do, especially combined
with something like python's virtualenv,

and similar concepts exist for other
languages, where you can sort of say

I want to (NPM does the same thing as well
with its node modules, for example) where

I'm gonna compile the dependencies of
this package in sort of a subdirectory

of its own, and all of the versions that it
requires are going to be built in there

and you can do this separately for separate
projects so there they have

different dependencies or the same dependencies
with different versions

they still sort of kept separate. And that
is one thing that's hard to achieve

with system packages

Next question

What's the easiest and best profiling tools
to use to improve performance of my code?

This is a topic we could talk
about for a very long time

The easiest and best is to print stuff using time

Like, I'm not joking, very often
the easiest thing is in your code

At the top you figure out what the current
time is, and then you do sort of

a binary search over your program of add
a print statement that prints how much

time has elapsed since the start of your
program and then you do that until you

find the segment of code that took the
longest. And then you go into that

function and then you do the same thing
again and you keep doing this until you

find roughly where the time was spent. It's
not foolproof, but it is really easy

and it gives you good information quickly

if you do need more advanced information
Valgrind has a tool called cache-grind?

call grind? Cache grind? One of the two.

and this tool lets you run your program and

measure how long everything takes and
all of the call stacks, like which

function called which function, and what
you end up with is a really neat

annotation of your entire program source
with the heat of every line basically

how much time was spent there. It does
slow down your program by like an order

of magnitude or more, and it doesn't really
support threads but it is really

useful if you can use it. If you can't,
then tools like perf or similar tools

for other languages that do usually some
kind of sampling profiling like we

talked about in the profiler lecture, can
give you pretty useful data quickly,

but it's a lot of data around
this, but they're a little bit

biased and what kind of things they usually
highlight as a problem and it

can sometimes be hard to extract meaningful
information about what should

I change in response to them. Whereas the
sort of print approach very quickly

gives you like this section
of code is bad or slow

I think would be my answer

Flamegraphs are great, they're a good way
to visualize some of this information

Yeah I just have one thing to add,
oftentimes programming languages

have language specific tools for profiling
so to figure out what's the

right tool to use for your language like if
you're doing JavaScript in the web browser

the web browser has a really nice tool for
doing profiling you should just use that

or if you are using go, for example, go has a built-in
profiler is really good you should just use that

A last thing to add to that

Sometimes you might find that doing this binary
search over time that you're kind of

finding where the time is going, but this
time is sometimes happening because

you're waiting on the network, or you're
waiting for some file, and in that case

you want to make sure that the time
that is, if I want to write

like 1 gigabyte file or like read 1
gigabyte file and put it into memory

you want to check that the actual time
there, is the minimum amount of time

you actually have to wait. If it's ten times
longer, you should try to use some

other tools that we covered in the debugging
and profiling section to see

why you're not utilizing all your
resources because that might...

Because that might be a lot of what's happening
thing, like for example, in my research

in machine learning workloads, a lot of
time is loading data and you have to

make sure well like the time it takes to
load data is actually the minimum amount

of time you want to have that happening

And to build on that, there are actually
specialized tools for doing things like

analyzing wait times. Very often when
you're waiting for something what's

really happening is you're issuing your
system call, and that system call takes

some amount of time to respond. Like you do
a really large write, or a really large read

or you do many of them, and one thing
that can be really handy here is

to try to get information out of the
kernel about where your program is

spending its time. And so there's (it's
not new), but there's a relatively

newly available thing called BPF or eBPF.
Which is essentially kernel tracing

and you can do some really cool things with
it, and that includes tracing user programs.

It can be a little bit awkward to
get started with, there's a tool

called BPF trace that i would recommend
you looking to, if you need to do like

this kind of low-level performance debugging.
But it is really good for this

kind of stuff. You can get things like
histograms over how much time was spent

in particular system calls

It's a great tool

What browser plugins do you use?

I try to use as few as I can get away with using

because I don't like things being in
my browser, but there are a couple of

ones that are sort of staples.
The first one is uBlock Origin.

So uBlock Origin is one of many ad blockers but
it's a little bit more than an ad blocker.

It is (a what do they call it?) a
network filtering tool so it lets

you do more things than just block ads.
It also lets you like block connections

to certain domains, block connections
for certain types of resources

So I have mine set up in what they call
the Advanced Mode, where basically

you can disable basically all network requests.
But it's not just Network requests,

It's also like I have disabled all inline
scripts on every page and all

third-party images and resources, and then
you can sort of create a whitelist

for every page so it gives you really
low-level tools around how to

how to improve the security of your browsing.
But you can also set it in not the

advanced mode, and then it does much of
the same as a regular ad blocker would

do, although in a fairly efficient way
if you're looking at an ad blocker it's

probably the one to use and it
works on like every browser

That would be my top pick I think,

I think probably the one I
use like the most actively

is one called Stylus. It lets you modify
the CSS or like the stylesheets

that webpages have. And it's pretty
neat, because sometimes you're

looking at a website and you want
to hide some part of the website

you don't care about. Like maybe a ad, maybe
some sidebar you're not finding useful

The thing is, at the end of
the day these things are

displaying in your browser, and you
have control of what code is

executing and similar to what Jon was
saying, like you can customize this

to no end, and what I have for a lot of
web pages like hide this this part, or

also trying to make like dark modes for
them like you can change pretty much the

color for every single website. And what
is actually pretty neat is that there's

like a repository online of people that
have contributed this is stylesheets

for the websites. So someone probably
has (done) one for GitHub

Like I want dark GitHub and someone has
already contributed one that makes

that much more pleasing to browse. Apart
from that, one that it's not really

fancy, but I have found incredibly helpful
is one that just takes a screenshot an

entire website. And It will
scroll for you and make

compound image of the entire website and that's
really great for when you're trying to

print a website and is just terrible.

(It's built into Firefox)

oh interesting

oh now that you mention builtin to Firefox,
another one that I really like about

Firefox is the multi account containers

(Oh yeah, it's fantastic)

Which kind of lets you

By default a lot of web browsers, like
for example Chrome, have this

notion of like there's session that you
have, where you have all your cookies

and they are kind of all shared from the
different websites in the sense of

you keep opening new tabs and unless you go into
incognito you kind of have the same profile

And that profile is the same for
all websites, there is this

Is it an extension or is it built in?

(it's a mix, it's complicated)

So I think you actually have to say you want
to install it or enable it and again

the name is Multi Account Containers and
these let you tell Firefox to have

separate isolated sessions. So
for example, you want to say

I have a separate sessions for whenever I
visit to Google or whenever I visit Amazon

and that can be pretty neat, because then you can

At a browser level it's ensuring that no information
sharing is happening between the two of them

And it's much more convenient than
having to open a incognito window

where it's gonna clean all the time the stuff

(One thing to mention is Stylus vs Stylish)

Oh yeah, I forgot about that

One important thing is the browser extension
for side loading CSS Stylesheets

it's called a Stylus and that's different
from the older one that was

called Stylish, because that one got
bought at some point by some shady

company, that started abusing it not only to have

that functionality, but also to read your
entire browser history and send that

back to their servers so they could data mine it.

So, then people just built this open-source alternative
that is called Stylus, and that's the one

we recommend. Said that, I think the repository
for styles is the same for the

two of them, but I would have
to double check that.

Do you have any browser plugins Anish?

Yes, so I also have some recommendations
for browser plugins

I also use uBlock Origin and I also use Stylus,

but one other one that I'd recommend is
integration with a password manager

So this is a topic that we have in
the lecture notes for the security

lecture, but we didn't really get to talk
about in detail. But basically password

managers do a really good job of increasing
your security when working

with online accounts, and having browser
integration with your password manager

can save you a lot of time like you
can open up a website then it can

autofill your login information for you
sir and you go and copy and paste it

back and forth between a separate program
if it's not integrated with your

web browser, and it can also, this integration,
can save you from certain

attacks that would otherwise be possible if
you were doing this manual copy pasting.

For example, phishing attacks. So
you find a website that looks very

similar to Facebook and you go to log in
with your facebook login credentials and

you go to your password manager and copy
paste the correct credentials into this

funny web site and now all of a sudden
it has your password but if you have

browser integration then the extension
can automatically check

like. Am I on F A C E B O O K.com,or
is it some other domain

that maybe look similar and it will not enter
the login information if it's the wrong domain

so browser extension for
password managing is good

Yeah I agree

Next question

What are other useful data wrangling tools?

So in yesterday's lecture, I mentioned curl, so
curl is a fantastic tool for just making web

requests and dumping them to your terminal.
You can also use it for things

like uploading files which is really handy.

In the exercises of that lecture we also talked about
JQ and pup which are command line tools that let you

basically write queries over JSON
and HTML documents respectively

that can be really handy. Other
data wrangling tools?

Ah Perl, the Perl programming language is

often referred to as a write only
programming language because it's

impossible to read even if you wrote it.
But it is fantastic at doing just like

straight up text processing, like nothing
beats it there, so maybe worth learning

some very rudimentary Perl just
to write some of those scripts

It's easier often than writing some like hacked-up
combination of grep and awk and sed,

and it will be much faster to just tack something
up than writing it up in Python, for example

but apart from that, other data wrangling

No, not off the top of my head really

column -t, if you pipe any white space separated

input into column -t it will align all
the white space of the columns so that

you get nicely aligned columns that's, and
head and tail but we talked about those

I think a couple of additions to that,
that I find myself using commonly

one is vim. Vim can be pretty useful
for like data wrangling on itself

Sometimes you might find that the operation
that you're trying to do is

hard to put down in terms of piping
different operators but if you

can just open the file and just record

a couple of quick vim macros to do what you
want it to do, it might be like much,

much easier. That's one, and then the other
one, if you're dealing with tabular

data and you want to do more complex operations
like sorting by one column,

then grouping and then computing some sort
of statistic, I think a lot of that

workload I ended up just using Python
and pandas because it's built for that

And one of the pretty neat features that
I find myself also using is that it

will export to many different formats.
So this intermediate state

has its own kind of pandas dataframe
object but it can

export to HTM, LaTeX, a lot of different
like table formats so if your end

product is some sort of summary table, then pandas
I think it's a fantastic choice for that

I would second the vim and also
Python I think those are

two of my most used data wrangling tools.
For the vim one, last year we had a demo

in the series in the lecture notes, but
we didn't cover it in class we had a

demo of turning an XML file into a JSON version
of that same data using only vim macros

And I think that's actually the
way I would do it in practice

I don't want to go find a tool that does
this conversion it is actually simple

to encode as a vim macro,
then I just do it that way

And then also Python especially in an interactive
tool like a Jupyter notebook

is a really great way of doing data wrangling

A third tool I'd mention which
I don't remember if we

covered in the data wrangling
lecture or elsewhere

is a tool called pandoc which can do transformations
between different text

document formats so you can convert from
plaintext to HTML or HTML to markdown

or LaTeX to HTML or many other formats
it actually it supports a large

list of input formats and a
large list of output formats

I think there's one last one which I mentioned briefly
in the lecture on data wrangling which is

the R programming language, it's
an awful (I think it's an awful)

language to program in. And i would never
use it in the middle of a data wrangling

pipeline, but at the end, in order to like produce
pretty plots and statistics R is great

Because R is built for doing
statistics and plotting

there's a library for are called
ggplot which is just amazing

ggplot2 i guess technically It's
great, it produces very

nice visualizations and it lets you do,
it does very easily do things like

If you have a data set that has like multiple
facets like it's not just X and Y

it's like X Y Z and some other variable,
and then you want to plot like the

throughput grouped by all of those parameters
at the same time and produce

a visualization. R very easily let's you
do this and I haven't seen anywhere

that lets you do that as easily

Next question,

What's the difference between
Docker and a virtual machine

What's the easiest way to explain this? So docker

starts something called containers and
docker is not the only program that

starts containers. There are many others
and usually they rely on some feature of

the underlying kernel in the case of
docker they use something called LXC

which are Linux containers and the basic
premise there is if you want to start

what looks like a virtual machine that
is running roughly the same operating

system as you are already running on your
computer then you don't really need

to run another instance of the kernel
really that other virtual machine can

share a kernel. And you can just use the
kernels built in isolation mechanisms to

spin up a program that thinks it's
running on its own hardware but in

reality it's sharing the kernel and so this
means that containers can often run

with much lower overhead than a full virtual
machine will do but you should

keep in mind that it also has somewhat weaker
isolation because you are sharing

a kernel between the two if you spin up
a virtual machine the only thing that's

shared is sort of the hardware and to
some extent the hypervisor, whereas

with a docker container you're sharing
the full kernel and the that is a

different threat model that you
might have to keep in mind

One another small note there as Jon pointed
out, to use containers something

like Docker you need the underlying operating
system to be roughly the same

as whatever the program that's running
on top of the container expects and so

if you're using macOS for example, the
way you use docker is you run Linux

inside a virtual machine and then you can
run Docker on top of Linux so maybe

if you're going for containers in order
to get better performance your trading

isolation for performance if you're running
on Mac OS that may not work out

exactly as expected

And one last note is that there
is a slight difference, so

with Docker and containers,
one of the gotchas you have

to be familiar with is that containers
are more similar to virtual

machines in the sense of that they will
persist all the storage that you

have where Docker by default won't have that.

Like Docker is supposed to be running

So the main idea is like I want
to run some software and

I get the image and it runs and if you
want to have any kind of persistent

storage that links to the host system
you have to kind of manually specify

that, whereas a virtual machine is using
some virtual disk that is being provided

Next question

What are the advantages of each operating system

and how can we choose between them?
For example, choosing the best Linux

distribution for our purposes

I will say that for many, many tasks the

specific Linux distribution that you're
running is not that important

the thing is, it's just what kind of

knowing that there are different types
or like groups of distributions,

So for example, there are some distributions
that have really frequent updates

but they kind of break more easily. So for
example Arch Linux has a rolling update

way of pushing updates, where things might
break but they're fine with the things

being that way. Where maybe where you
have some really important web server

that is hosting all your business
analytics you want that thing

to have like a much more steady way of
updates. So that's for example why you

will see distributions like Debian being

much more conservative about what they push, or
even for example Ubuntu makes a difference

between the Long Term Releases
that they are only update every

two years and the more periodic
releases of one there is a

it's like two a year that they make.
So, kind of knowing that there's the

difference apart from that some distributions
have different ways

of providing the binaries
to you and the way they

have the repositories so I think a lot of Red
Hat Linux don't want non free drivers in

their official repositories where I
think Ubuntu is fine with some of

them, apart from that I think like just
a lot of what is core to most Linux

distros is kind of shared between them
and there's a lot of learning in the

common ground. So you don't have
to worry about the specifics

Keeping with the theme of this class being somewhat
opinionated, I'm gonna go ahead and say

that if you're using Linux especially for
the first time choose something like

Ubuntu or Debian. So you Ubuntu to is a
Debian based distribution but maybe is a

little bit more friendly, Debian is a little
bit more minimalist. I use Debian

and all my servers, for example. And I use
Debian desktop on my desktop computers

that run Linux if you're going for maybe
trying to learn more things and you want

a distribution that trades stability for
having more up-to-date software maybe

at the expense of you having to fix a
broken distribution every once in a

while then maybe you can consider something
like Arch Linux or Gentoo

or Slackware. Oh man, I'd say that like
if you're installing Linux and just like

want to get work done Debian is a great choice

Yeah I think I agree with that.

The other observation is like
you couldn't install BSD

BSD has gotten, has come a long way from
where it was. There's still a bunch of

software you can't really get for BSD but
it gives you a very well-documented

experience and and one thing that's different
about BSD compared to Linux is

that in an BSD when you install BSD you
get a full operating system, mostly

So many of the programs are maintained by
the same team that maintains the kernel

and everything is sort of upgraded together,
which is a little different

than how thanks work in the Linux world it does

mean that things often move a little bit
slower. I would not use it for things

like gaming either, because drivers support
is meh. But it is an interesting

environment to look at. And then for things
like Mac OS and Windows I think

If you are a programmer, I don't know why
you are using Windows unless you are

building things for Windows; or you want
to be able to do gaming and stuff

but in that case, maybe try dual booting,
even though that's a pain too

Mac OS is a is a good sort of middle point
between the two where you get a system

that is like relatively nicely polished
for you. But you still have access to

some of the lower-level bits
at least to a certain extent.

it's also really easy to dual boot Mac OS and Windows
it is not quite the case with like Mac OS and

Linux or Linux and Windows

Alright, for the rest of the
questions so these are

all 0 upvote questions so maybe we can go
through them quickly in the last five

or so minutes of class. So the next
one is Vim versus Emacs? Vim!

Easy answer, but a more serious answer is like I think
all three of us use vim as our primary editor

I use Emacs for some research specific
stuff which requires Emacs but

at a higher level both editors have interesting
ideas behind them and if you

have the time is worth exploring both
to see which fits you better and also

you can use Emacs and run it in a vim
emulation mode. I actually know a

good number of people who do that so
they get access to some of the cool

Emacs functionality and some of the cool
philosophy behind that like Emacs is

programmable through Lisp which is kind of cool.

Much better than vimscript, but people like
vim's modal editing, so there's an

emacs plugin called evil mode which gives
you vim modal editing within Emacs so

it's not necessarily a binary choice you
can kind of combine both tools if you

want to. And it's worth exploring
both if you have the time.

Next question

Any tips or tricks for machine
learning applications?

I think, like knowing how

a lot of these tools, mainly the data wrangling

a lot of the shell tools, it's really
important because it seems a lot

of what you're doing as machine learning
researcher is trying different things

but I think one core aspect of doing that,
and like a lot of scientific work is being

able to have reproducible results
and logging them in a sensible way

So for example, instead of trying to come
up with really hacky solutions of how

you name your folders to make
sense of the experiments

Maybe it's just worth having for example

what I do is have like a JSON
file that describes the

entire experiment I know like all the parameters
that are within and then I can

really quickly, using the tools that
we have covered, query for all the

experiments that have some specific
purpose or use some data set

Things like that. Apart from that, the other
side of this is, if you are running

kind of things for training machine
learning applications and you

are not already using some sort of
cluster, like university or your

company is providing and you're just kind
of manually sshing, like a lot of

labs do, because that's kind of the easy way

It's worth automating a lot of that job
because it might not seem like it but

manually doing a lot of these operations
takes away a lot of your time and also

kind of your mental energy
for running these things

Anymore vim tips?

I have one. So in the vim lecture we tried
not to link you to too many different

vim plugins because we didn't want that
lecture to be overwhelming but I think

it's actually worth exploring vim plugins
because there are lots and lots

of really cool ones out there.
One resource you can use is the

different instructors dotfiles like a lot
of us, I think I use like two dozen

vim plugins and I find a lot of them quite
helpful and I use them every day

we all use slightly different subsets of
them. So go look at what we use or look

at some of the other resources we've linked
to and you might find some stuff useful

A thing to add to that is, I don't think
we went into a lot detail in the

lecture, correct me if I'm wrong. It's
getting familiar with the leader key

Which is kind of a special key
that a lot of programs will

especially plugins, that will link to
and for a lot of the common operations

vim has short ways of doing it, but you
can just figure out like quicker

versions for doing them. So for example, like
I know that you can do like semicolon WQ

to save and exit or that you
can do like capital ZZ but I

just actually just do leader (which for
me is the space) and then W. And I have

done that for a lot of a lot of kind of
common operations that I keep doing all

the time. Because just saving one keystroke
for an extremely common operation

is just saving thousands a month

Yeah just to expand a little bit

on what the leader key is so in vim you
can bind some keys I can do like ctrl J

does something like holding one key and
then pressing another I can bind that to

something or I can bind a single keystroke
to something. What the leader

key lets you do, is bind

So you can assign any key
to be the leader key and

then you can assign leader followed by
some other key to some action so for

example like Jose's leader key is space
and they can combine space and then

releasing space followed by some other
key to an arbitrary vim command so it

just gives you yet another way of binding
like a whole set of key combinations.

Leader key plus kind of any key on
the keyboard to some functionality

I think I've I forget whether
we covered macros in the vim

uh sure but like vim macros are worth
learning they're not that complicated

but knowing that they're there and knowing
how to use them is going to save

you so much time. The other one is something
called marks. So in vim you can

press m and then any letter on your keyboard
to make a mark in that file and

then you can press apostrophe on the
same letter to jump back to the same

place. This is really useful if you're
like moving back and forth

between two different parts of your code
for example. You can mark one as A and

one as B and you can then jump between
them with tick A and tick B.

There's also Ctrl+O which jumps to the previous
place you were in the file no matter

what caused you to move. So for example
if I am in a some line and then I jump

to B and then I jump to A, Ctrl+O will
take me back to B and then back to the

place I originally was. This can also be
handy for things like if you're doing a

search then the place that you
started the search is a part of

that stack. So I can do a search I can
then like step through the results

and like change them and then Ctrl+O
all the way back up to the search

Ctrl+O also lets you move across files so
if I go from one file to somewhere else in

different file and somewhere else in the
first file Ctrl+O will move me back

through that stack and then there's
Ctrl+I to move forward in that

stack and so it's not as though you
pop it and it goes away forever

The command colon earlier is really handy.
So, colon earlier gives you an earlier

version of the same file and it it does
this based on time not based on actions

so for example if you press a bunch of like
undo and redo and make some changes

and stuff, earlier will take a literally
earlier as in time version of your file

and restore it to your buffer. This can
sometimes be good if you like undid and

then rewrote something and then realize
you actually wanted the version that was

there before you started undoing earlier
let's you do this. And there's a plug-in

called undo tree or something like
that There are several of these,

that let you actually explore the full
tree of undo history the vim keeps

because it doesn't just keep a linear history
it actually keeps the full tree

and letting you explore that might in
some cases save you from having to

re-type stuff you typed in the past or
stuff you just forgot exactly what you

had there that used to work and no longer
works. And this is one final one I

want to mention which is, we mentioned
how in vim you have verbs and nouns

right to your verbs like delete or yank
and then you have nouns like next of

this character or percent to swap brackets
and that sort of stuff the

search command is a noun so you can do
things like D slash and then a string

and it will delete up to the next match
of that pattern this is extremely useful

and I use it all the time

One another neat addition on the undo stuff
that I find incredibly valuable in

an everyday basis is that like one of
the built-in functionalities of vim

is that you can specify an undo directory
and if you have a specified an

undo directory by default vim, if you
don't have this enabled, whenever you

enter a file your undo history is
clean, there's nothing in there

and as you make changes and then
undo them you kind of create this

history but as soon as you exit the
file that's lost. Sorry, as soon

as you exit vim, that's lost. However
if you have an undodir, vim is

gonna persist all those changes into
this directory so no matter how many

times you enter and leave that history
is persisted and it's incredibly

helpful because even like

it can be very helpful for
some files that you modify

often because then you can kind of keep
the flow. But it's also sometimes really

helpful if you modify your bashrc see and
something broke like five days later and

then you've vim again. Like what actually
did I change ,if you don't

have say like version control, then
you can just check the undos and

that's actually what happened. And
the last one, it's also really

worth familiarizing yourself with registers
and what different special

registers vim uses. So for example if
you want to copy/paste really that's

gone into in a specific register and if you
want to for example use the a OS a copy

like the OS clipboard, you should
be copying or yanking

copying and pasting from a different register
and there's a lot of them and yeah

I think that you should explore, there's
a lot of things to know about registers

The next question is asking about two-factor
authentication and I'll just give

a very quick answer to this one in the interest
of time. So it's worth using two

factor auth for anything security sensitive
so I use it for my GitHub

account and for my email and stuff like
that. And there's a bunch of different

types of two-factor auth. From SMS based
to factor auth where you get special

like a number texted to you when you try
to log in you have to type that number

and to other tools like universal to
factor this is like those Yubikeys

that you plug into your you have
to tap it every time you login

so not all, (yeah Jon is holding a
Yubikey), not all two-factor auth is

created equal and you really want to be
using something like U2F rather than SMS

based to factor auth. There something
based on one-time pass codes that you

have to type in we don't have time to get
into the details of why some methods

are better than others but at a high
level use U2F and the Internet has

plenty of explanations for why other
methods are not a great idea

Last question, any comments on differences
between web browsers?

Yes

Differences between web browsers, there
are fewer and fewer differences between

web browsers these day. At this point
almost all web browsers are chrome

Either because you're using Chrome or
because you're using a browser that's

using the same browser engine as Chrome.
It's a little bit sad, one might say, but

I think these days whether you choose

Chrome is a great browser for security reasons

if you want to have something
that's more customizable or

you don't want to be tied to Google then
use Firefox, don't use Safari it's a

worse version of Chrome. The new Internet
Explorer edge is pretty decent and also

uses the same browser engine as
Chrome and that's probably fine

although avoid it if you can because it
has some like legacy modes you don't

want to deal with. I think that's

Oh, there's a cool new browser called flow

that you can't use for anything useful
yet but they're actually writing

their own browser engine and that's really neat

Firefox also has this project called servo which is
they're really implementing their browser engine

in Rust in order to write it to be like
super concurrent and what they've done

is they've started to take modules
from that version and port them

over to gecko or integrate them with gecko
which is the main browser engine

for Firefox just to get those
speed ups there as well

and that's a neat neat thing
you can be watching out for

That is all the questions, hey we did it. Nice

I guess thanks for taking the missing semester
class and let's do it again next year