Lecture 10: Potpourri (2020)

Hi, can everyone hear me okay?

Okay, so welcome back to

the missing semester of your CS education,

today we're having as a
lecture topic "potpourri"

It's gonna be some miscellaneous
combination of topics

that we the instructors find
that are interesting

But none of them were on their own lecture

because they're certain topics that
we just want you to know about

because they can be really helpful

And since we're not gonna delve into
a lot of detail in the topics

If you're more interested about them just
feel free to come and ask us questions

at the end or as we go over
them in the lecture

So the first thing I want to talk about
is keyboard re-mappings

So by now you probably realized that

we have encouraged you to use the
keyboard as your main input method

so for example, when we went
into the editors lecture,

one of the main ideas of Vim was using
your keyboard as much as possible

so you don't have to rely on going to the
mouse because going to the mouse is slow

And the thing is your keyboard, as with
many things in your computer

is nothing kind of magical,
it can be configured

And it's worth configuring because a lot
of the defaults might not be optimal

The most simple modification that you
can do is just to remap keys

So one of the things we alerted
in the editor lecture

is the Caps Lock key is a really good key

because it's kind of right
in the home row

and it's kind of large and there,
but it's not useful

You'll probably realize that you don't
use your Caps Lock as often

as when you'd want to use that

So you can just remap your
Caps Lock key

to something more useful as we mentioned

like Escape if you're a Vim user

or like Control if you're an Emacs user

or useful re-mappings, like a lot of the

upper row 'F' function keys,
or like Print Screen

You can remap them for example
to your media key

so like when you type Print Screen,

you probably don't have to
do Print Screen that often

but you probably want to play
or pause your music

And pretty much every operating system has

some tools that you can use
to configure this

I'm not gonna go into the details

but there's some of them
listed in the notes

Erm... Let me check

Oh yeah, another thing that you can
do with keyboard re-mappings is

that you can do more complex combinations

You can have a combination of keys
mapped to some action

So for example, I have keyboard
re-mappings that

whenever I do Ctrl+Enter, I open
a new terminal window

because that's a thing I do
fairly often and

by default on Mac there is no
key binding to do that

or Ctrl+Shift+Enter will open a
new browser window,

another operation that I do
on a daily basis

So I don't have to grab my mouse
and go to Chrome to do that

You can also do remapping to
perform actions

If you don't want to be typing your
password all the...

Sorry, not your password

Your Email or your password,
or for example, your MIT ID

like, you may not remember it by heart

then you can just have a
keyboard combination that

will just perform the action of
pasting that text

Lastly, there are more...

Right now it looks like you just have
to do some function of

"this is the keys that you press and
this is the action that happens"

But actually there are more complex
keyboard combinations, and

as you go through you'll learn
so you can do keyboard sequences

So for example when we were
dealing with tmux,

in tmux there was this notion of

first you press Ctrl+A or Ctrl+B, like
you press some prefix

and then some other key and
that means something

A lot of these softwares allow that

So for example, in my keyboard, since
I'm not using Caps Lock at all, but

every so often I have to use when my
undergrad has some software that

relied on Caps Lock for changing modes

Then I can press Shift five times
in a row quickly, and then

this software that is in the middle
interpreting these commands and

remapping to some other, will send a
single Caps Lock command for that

Some more examples of that is that

I mentioned that you can use your
Caps Lock key to map to Escape or Control

but you actually can remap it to both

So in my computer when

I just tap the Caps Lock key, that's
interpreted as an Escape

However if I press it and hold it,
this software can understand the

difference between quickly pressing it and

just holding it for using in
combination with some other key

and then in that case
it's mapped to Control

So a lot of these more
advanced configurations are

supported on a lot of these tools

As I mentioned we have a
short list of good defaults

for these programs for
Windows, Mac OS and Linux

Any questions on this topic?

Okay, now I'm going to cover an
unrelated topic to keyboard mappings

[chuckles]

We're going to see a lot of these
unrelated transitions in this lecture

And it's the concept of "daemons"

So probably you have...

Maybe if you are not familiar with
the world, it might seem alien,

but the concept of daemon
you're pretty familiar with

Most computers when you are running them

there's software that you start and run,

like the commands that we have been seeing

You'd type "ls" and then you're
calling the List command

The "ls" command executes because you
asked it to execute and then it finishes

But a lot of other programs are just
running as background processes

and they're just executing
in the background

and waiting for events to happen

or enabling some sort of functionality
in your computer

Examples of these processes may be
like your Network Man[ager],

like the part of your computer
that is managing the network

or the part of your computer
that is managing the display

Things like that

You will see that a lot of what
is enabled by daemons is

usually programs that end with a 'd'

So for instance, when you are
SSHing into a computer,

the receiving computer has to
have a SSH daemon

and the program is called "sshd"

and if this program is not running then

there's no way for me
to SSH into the computer

If the program is running, then
that program will be listening

and when you do SSH to that server,

then some incoming request
is gonna enter the computer

the computer is gonna send it to this
daemon that is running in the background

and then the daemon is gonna check
whether you have authorization,

and if so, it's gonna start some login
shell that you can start executing from

And different OSes handle this
somewhat differently

The main idea is they all have
some sort of system daemon

that responds a lot of
these smaller daemons

In Linux, which is one of the OSes that
we are choosing for lot of the examples,

the tool that you're using
is the "systemd"

Again, first the system daemon is gonna
start a lot of these processes

and if you use this "systemctl" command,

you can check for the status
of different daemons

you can check for which ones are running,

you can ask it to start processes,
to stop them

This is kind of an once-off operation

You can also enable it
and then disable them,

which will tell the system
to run them at boot

or stop running the boot
if they were enabled

And perhaps more interestingly, you can
configure your own "systemd" units

So, so far all the examples are a lot
of what the computer has to do,

but say you want to run a web server.
One solution, you could just like,

every time you start your computer
you could open a tmux session

and then execute the command,

but that's not really the way that your
computer expects daemons to be run

The way your computer expects
daemons to be run is

by using some sort of "systemd" unit

It's like a configuration that tells
"systemd" how to execute this process

So an example of this is...

Here's a very simple example

So what is happening here is

we're describing to "systemd" what needs
to be done for this program to execute.

This example is just running
a simple Python app

You can think of it as a web server that
can be implemented using

some Python web server library.
And here we're saying

this is the description, we're
saying after, like this is important

"Systemd" has a list of
services that have to start

Like, all these daemons have to be started

but maybe there are dependencies
between these daemons, so here

we're saying "no, you should only start
this after the network has been set up"

because otherwise how will you even try to

configure our web server if I cannot
listen to a network port?

And then we are defining what
users should run this, because

you may want to run this as
your user, or maybe other user,

or maybe the root user
should be running this,

and then what command to run
and under what directory

And whenever you have this,

there can be small corner cases that
you might have to debug, but

this is kind of the core idea and
it can be really useful to

automate the process of
running processes in the background

A small side note to this is
the fact that if you just want

to run a command every so often,
like in some periodicity,

say every morning I want to
do something in my computer,

you could write a daemon that just does
something and then sleeps for a day, but

actually Linux and Mac OS have already a
daemon that does this called "crond"

and "crond" will take another type of
configuration file where you can say

oh, I want to run a command
every day at 8 AM,

or I want to run a command
every 5 minutes,

and it will just check for
this event and execute it

And with a lot of things
you will find that

there are already daemons that
have been configured for that

Any questions regarding daemons?

[student talking indistinctly]

So the question is whether there is

a folder in the computer
where all of these are

So yeah, like yes and no

Some of these configuration files are
in a couple of different folders

depending on whether they are
system daemons or they are user daemons

Here you can see at the very first line
is where you will place this

for the system daemon to recognize
that it has been installed

but if you just want to list
all the daemons that are running,

in Linux for example, you can just do
"systemctl status"

and that's gonna print
a tree of all the systems

and which daemon was spawned
by which other daemon

and a lot of them will be
spawned directly by "systemd"

The next topic is going to be
file systems in user space

So, kind of a quick intro to this is the
fact that whenever you're using a modern..

Oh, yeah, sorry

And whenever you're using a
modern operating system,

you are not tied to a specific file system

So modern systems are fairly modular
and you can for example,

in Linux there are different file systems
that you can use, and

the way this works is because the kernel,
which is what is running most of the

operating system, has some modules that
know how to interact with a file system

So usually when you do something
like "touch foobar",

this is happening at a user level

and then this is going through
to the kernel level

and there is some kind of
layer here that is checking

where this action is happening to
figure out what file system it is under

So for example, you will
have multiple disks

and all the different disks
have different file systems,

so the kernel has to figure out which
file system operations to use,

and say this file might be in an "Ext4",
which is the most common Linux one,

then whenever you do "touch foobar",
the kernel will hear that

and then it will try to figure out like,
oh, this lives in an Ext4 file system and

it will perform the associated instruction
for creating a file in an Ext4 file system

However, the caveat of having
a system like this is

right now I cannot have user code
that defines how to create a file

and that might be kind of
useful in some cases

Say I want to have a file system that
every time someone creates a file,

it sends me an email, so I can know that
people are creating these files

Here I cannot modify
the kernel to add this

So the solution to this is
something called "FUSE"

And FUSE is a way of having
file systems in user space

So what FUSE will do is,

if this file instead of being in Ext4,
if this file is in a FUSE file system,

FUSE will forward this operation to
some other part of user call

that will say "oh, create this file"

And here I can have the part of the code
that sends an email to me,

saying "oh, this file has been created"

And in case you want to
still create the file,

it can forward back the request
to do some more kernel operations

It might not seem really practical,
but this is just the theory

In practice, why this is useful is because

now you can have user level code
that executes arbitrary actions

when you try to perform
file system operations

A really Interesting example of this
is called "SSHFS"

On an SSHFS FUSE file system,

whenever you try to create,
open, read, write to a file,

instead of trying to do that
to a local file,

it has an SSH connection
to a remote server

So if I try to create a file here,
It will use that SSH connection to

forward that operation to
the remote system

and then it will perform it there

So to all my local computer, to the rest
of the programs running in my computer,

there is this path that looks
like it is here, but

all the operations that are performed
to the path are forwarded

to the remote file system

And with this idea, you will get
some examples in the notes,

and you will find more online,
of ways people have

leveraged this capability to do
fairly interesting file systems

So for example, if instead of having SSH,

you don't care about SSH because
you use Dropbox or Google Drive,

it's fine, people have implemented
FUSE file systems that will mount locally

and every time you try to do
an operation locally,

actually it goes to one of these
cloud storage providers

so you can also use something like
Amazon S3, or Google Cloud Storage,

that don't have the same
kind of UI system

that we synchronize, as
Dropbox or Google Drive

Another application of this that is not
related to doing something remotely,

is something like
an encrypted file system

You may have a file system that
every time you try to write to a file,

you will try to write it in plain text,
but it will capture that operation

it will encrypt on the go,

and then it will save it as a
regular file in your file system,

but that's actually encrypted

And once you dismount the file system,
once you remove the FUSE connection,

all that is left in your computer are just
regular files that are encrypted

The last topic I wanna cover is backups
and some good practices about them

The main idea is that for every
file that you care about,

if you don't have a backup of that file,

if you don't have a backup
stored of that file,

you can pretty much lose it at any moment

There are many different failure scenarios

One of them is just hardware failure.
So your hard drive can fail at any moment

So if you are just making a copy of your
files in the same drive, that's not useful

If your hard drive fails,
the files are gone

The same goes if you have an external
drive where you are making a copy,

but if you are storing everything in your
home, and your home burns down...

Which yes, it's unlikely, but
if it happens you just lost all your data

So you'd have some sort of
off-site backup for having this solution

Another thing to take into account is that

synchronization or mirroring options
are not backups

So Google Drive, Dropbox
that I was mentioning,

they will just propagate whatever is
happening in your computer

This goes also for
hardware mirroring, like RAID

They are just making a copy

If you accidentally delete a file,

or someone maliciously deletes your files,
or encrypts them using some ransomware,

then you might have a copy, but
you have a copy of the same useless data

You actually have to have a solution of
how you're running your backups

And you should be asking yourself

what actually someone needs to know/have
about you in order to delete all your data

And we have linked different softwares
in the notes about how to do this

The last thing I want to mention about
backups is that a lot of the time

when you think about backups, you just
think about the local files

and like, all my photos and my tax return,
and how can I make a backup of that?

But increasingly in the modern age,
there are more and more web applications

and a lot of data might only live
in some cloud provider,

like for example if you have Webmail,

and you're not synchronizing it
to your computer,

It's only living in
that provider's servers

And if you don't have a copy for that

and for some reason you
lose access to that account

because you forgot your password,
you got hacked,

they think you have violated the
terms of service... All that data is gone

So you should look into some tools
that people have developed

for making offline copies of all that data

so you can make regular backups of that

And that kind of ends this
short section on backups

Any questions so far?

(student) When you said that
a hard drive can fail at any time

Is there a reason for it to fail?

[unintelligible]

Like if I have my external hard drive
sitting at my parents house or something

And my computer here, is that enough?
Or any drive can just fail?

(Jose) Any drive can fail at any moment.
Like we don't...

Different media have
different rates of failure

and there are really good
statistics online

So for example,

spinning hard drives have a higher
rate of failure than SSDs for example,

like Solid-State Drives

And what's another case?
Or like CD drives for example

But if you drop a hard drive it's a
higher rate of that failing, of course

But in general we don't really have
an end-all solution

for saying "this media is not gonna fail"

Like pretty much, like SD cards, SSDs,
hard drives, CDs, degrade with time

Pretty much every data is kind of
bound to this degradation

or like this fact that it could
be lost at any moment

And you should also know that
data can become corrupted,

your disk might look like it's okay,
but maybe some files were corrupted

and something like
synchronization techniques,

like Google Drive or Dropbox,
will propagate that corruption

And by the time that you realize that
things have gone wrong it's maybe too late

(Jon) Alright, we're gonna continue this trend of jumping between random topics and talk about APIs

So so far we've really been talking about how do you do things more efficiently locally on your computer?

Like I want to accomplish this task more efficiently. How do I like configure my editor? How do I use my shell?

But one thing you should realize is that very often you can integrate with the outside world as well

Most services that you interact with in your day-to-day provide some kind of API

For you to interact with the data that they store or the services that they provide

And usually those APIs are pretty well documented

If you looked at the APIs for things like Facebook, or Twitter, or Google Drive, or Gmail

Many of these have interfaces that you can interact with in order to use those services from your local machine

What's really neat is that you can often combine this with some of the stuff that we've talked about in lecture so far

Like for example in the Data Wrangling lecture, we looked at how you can create

These pipelines to extract data from some source that has a different format than you expected

So for instance, the US government has a free service

Where you can request the weather forecast for any given location in the US

And what you do is there is a URL that you request

And if you set the right parameters in that URL and then just fetch it, what you get back is JSON

Which is sort of a well-defined data format that you can then parse

And you can extract things like your 14-day weather forecast

And maybe you then pipe that into your shell and produce some kind of like

Handy alias in your terminal that's just gonna print...

Of some handy reference for the next 14 days of weather in whatever location you were in

These are things you can pretty easily construct and there's some notes...

There are some notes in the notes about how you might go about this

In general when you interact with these APIs, you're going to be using URLs of one form or another

And the exact format varies from service to service

But in general the URL is going to contain some set of parameters

But ultimately you're just gonna issue a web request to them and you're gonna get data back in some format

One command you should be aware of for interacting with these types of things is one called "curl"

So curl is a program that you invoke, you give it a URL

And it just fetches that URL and gives you back the response

What you do with that response is entirely up to you

Maybe you pipe it through a program like "gq". Sorry, "jq"

So jq is a JSON query tool that lets you take in data that's formatted as JSON

And then write a query over it to extract data that you're interested in

And this is one of the ways in which you can layer these tools to extract the data that you're interested in

Some of these services also require that you authenticate in one way or another

Like for example if you want to interact with a Facebook API

You need to have some authenticated token that proves who you are as far as Facebook is concerned

Otherwise, they can't say whether you're allowed to say, create a post as a given user

Very often these things are gonna use something called "OAuth", although not always

And you should look at the documentation for every service you care about

In general though you will get some kind of secret token back from the service

That you have to include in the requests that you make to them

Either in the URL or in additional sort of web headers, which you can also send with curl

Keep in mind though that these tokens are secret. They are another representation of your user

And anyone who gets their hand on them can basically pretend to be you

They can do whatever you can do with that token

So keep this in mind, don't stick them in your "dotfiles" and then push them onto GitHub

That will land you in trouble. You should think of them as a password

There are also really neat tools online for integrating services

So there's a service called "If This Then That"

Which basically provides integrations with a bunch of different services and lets you chain them together

And then also access them partially locally if you wish

This is something that's worth looking into

If there's a particular service you would like to interact with in a more efficient manner

Any questions about APIs?

Alright

Switching gears entirely, let's talk about command-line arguments

So command-line tools. There are a lot of them

And most of them take different arguments because they do different things

We've talked about looking at man pages for commands and that will tell you how this particular command works

What kind of flags and options you might give to it

And what it actually does when you invoke it

But there are some common themes that are useful to know about

Either in arguments that many programs take, or just concepts the many of them apply

The first of these we already mentioned a little bit in the lecture on command-line environments

Which is the "--help" flag. Very often you can pass this to a program and instead of running,

It will just print out information about how you can run this program, often in a very short, condensed way

A similar one is the "--version" flag, which just prints the version of the software you're using

This can be really handy if you're doing something like filing a bug report,

Which Anish is going to talk a little bit about later

And you want to report what version you're running on in case the bug has been fixed since

Often you can also do "-V" and that means the same as version. But again, check the man page

There's also "--verbose", or "-v" often

Which is a flag that lets you increase the output of the program

It makes the program print more about what it is doing. And very often you can repeat this flag

So you can do like "-vvvvv" in order to get more information from that tool

And this can be especially useful if you're trying to debug a problem

If you're like running "rsync" and you want to know why did it decide to copy this file?

Or why did it decide not to copy this file? That kind of debug output can be useful

And often there's a sort of an inverse flag called "quiet" or "silent"

Which means that the tool will not print anything unless it was like an error

Anything else, it will stay quiet about

Many tools, especially those that do destructive actions

Or some kind of action that you cannot undo

Provide what's known as a "dry run flag"

Exactly how this is represented in the command line varies from tool to tool

But essentially what this dry run mode will do is it will run the tool, but it will not actually make any changes

Instead it will just inform you of what it would have done if you hadn't run it with dry run

Many of these tools also have an interactive mode. So for example, th "rm" and "mv" tools both do

Often just "-i", although not always

When you run a tool in interactive mode

It will usually prompt you whenever it's about to do an action that you can't undo

And it will sort of prompt you for a confirmation that it should actually go ahead

When we're talking about destructive tools

Many of them are non-recursive by default

If you try to remove a directory or you try to operate on a full directory

They will not continue into the files inside of that directory

The reason being you might accidentally like, remove your entire hard drive and that seems bad

Therefore for many of these tools, they have a "recurse" flag

Often "-r", but again not always

Which lets them traverse down into the tree to go deeper, but you need to opt-in to this behavior

So this is for example, the case for "rm". This is also the case for "cp"

In many tools, when they ask you to give a file name or a path

And we talked about this a little bit in the data wrangling lecture

Instead of giving a file name, you can often just give a dash

Just a single "-", and what that means is standard input or standard output

Depending on whether that argument is an input file or an output file

This is handy to know about if you're trying to construct those kinds of data wrangling pipelines that we've talked about before

Many tools will also default to using standard in or standard out if you don't give any file name at all

Sometimes you want to pass something that looks like a flag or an option to a command

But you don't actually want it to be interpreted as a flag or an option

Consider for example, if you wanted to remove a file called "-i", what would you do?

Right, if you write the following command...

"rm -i"

Well, "-i" is a flag to "rm", so "rm" would, when you run this command say

Tell me what file to remove, you haven't given me a file

And it's because it interprets this as a flag

Similarly, if you do something like...

"ssh some machine, some command, and let's say, dash r"

So this is saying - run command "foo" on this machine over SSH, and I wanna pass "-r" to "foo"

Well, the way that both of these are gonna get interpreted is that these are flags

Or in this case, this is a flag. But to this command

Which is probably not what you expected

Actually in the case of SSH, it has some weird special behavior for some of these

But often if you want something to not be interpreted as a flag

There's a very simple way to opt-out of that, and that is using double dash

If you use double dash, what you tell the command

Is that verything following this, you should not interpret

So it will not be considered a flag or an option. In the case of "rm" you can do this

And now "rm" you will see that the first argument is a "--"

And then it will keep reading arguments, but it will not interpret them as flags

So when it gets the "-i", it will not interpret it as the dash i flag, but just as an argument dash i

Similarly, for SSH you can do this to indicate that these are both positional arguments

They are not flags or options and you should not interpret things that start with a dash

(student) But if you do like "--version", it's not gonna trigger that?

(Jon) No, so this is a " -- ", with a space on both sides

Any questions about any of this sort of command line conventions business?

Alright then let's talk about window managers

So most of you are used to some kind of drag-and-drop window manager

If you're running Windows, or Mac OS, or Ubuntu - what comes with the machine is like, there are windows and

They overlap partially on screen and you can like drag-and-drop and move them around and resize them and stuff

And that works fine

But it is not the only way to manage windows on your computer it turns out

So what you are used to is something called a floating window manager, but not all window managers are floating

Often you can opt-in to other types of window managers that have different behavior for how they arrange your desktop

A common alternative is a tiling window manager

So in a tiling window manager, rather than having floating windows,

Everything is set up into a tiled layout. When you start a program, its window is maximized

If you start another program, the original window shrinks in size

And then the new window takes up some subset of the total desktop space

At no time is your desktop background visible unless you have no programs open

All of the programs you have open on any given desktop are going to share that space

This looks a little bit like tmux panes, like we talked about earlier, where you can sort of split them in various directions

And one of the reasons why this is handy is it means you basically never need to go to your mouse

In order to move between windows, there are keyboard shortcuts to move to different windows

There are keyboard shortcuts for resizing the windows or swapping them around on screen

And this turns out to be a pretty efficient way to manage windows in your computer

Well, I won't go into too much of detail of what kind of window managers you might use

Just know that these exist out there and they're worth giving a look. They can be a lot more efficient to work with

Questions about window managers?

All right, VPNs. Totally related to the previous topic

So VPNs are like all the rage these days and this makes me really sad

It's not clear that VPNs are all the rage for any good reason

Because you should be aware of what a VPN does and does not get you

A VPN, in the best case, is really just a way for you to change your internet service provider

It's a way for you to make traffic on the Internet seem like it's coming from somewhere else than where you actually are

While that might seem attractive for certain purposes,

It's a little unclear what it buys you in terms of security

Because all you're really doing is shifting who you are trusting

Rather than trusting who is providing your current internet service,

You're trusting that whatever business is giving you that VPN service...

You're trusting that they, first of all, have set up this VPN business correctly,

But also that they are not tracking what you are doing

And it's not clear whether that change of trust is actually worth it

If you're sitting at some like dodgy public Wi-Fi network then maybe

But if you're sitting at MIT, it's not clear

Do you trust your VPN provider more than you trust MIT's IS&T?

Or maybe you do, but that is a decision that you need to make

About what you trust, who you trust and why?

You should also know that much of your traffic,

Especially the stuff that's on a sensitive nature on the Internet, is already encrypted

Whether that's sort of HTTPS or other protocols that use something like TLS, a lot of the sensitive data is already

Sent over encrypted channels and it doesn't really matter who your network provider is if you're on a dodgy Wi-Fi network

The stuff that matters is probably encrypted anyway. Might not be, but if it's not

Then your VPN provider can also see it in plain text just as much as whoever's hosting this dodgy Wi-Fi network

And notice that I said "in the best case above"

There are VPN providers who have been shown to be malicious, that do logging of all your traffic,

That sell that traffic to third parties. There are VPN providers that have forgotten to enable encryption on the VPN

All of these are real problems

And so you should think very carefully about whether a VPN actually serves any good purpose for you

Questions about VPNs? Yes?

(student) So I have a question about public Wi-Fi networks, because

The traffic from your computer to the router isn't encrypted between the computer and the router, right?

Except for what normally is via HTTPS and [unintelligible]

So then doesn't that mean that people could sniff out what domains I'm going to via the DNS request?

(Jon) So, it's a very good question

If you're on a public Wi-Fi network, then the traffic between you and the wireless access point is not encrypted

At least it's not encrypted sort of on the outer layer, but it might be encrypted in like HTTPS, for example

And it is totally true the people observing that Wi-Fi network will be able to see anything that is not encrypted

But the solution to that is to encrypt all your traffic, rather than necessarily going through a VPN

So one way to do this for example is to use DNS over TLS or DNS over HTTPS, which gives you

A way to actually encrypt even information that might otherwise leak in plain text

Rather than try to sort of trust some provider to do that for you

Now that said, in some cases you might have a trusted institution that provides a VPN network for you

So for example, MIT provides a VPN network for all MIT students and staff that you can sign up to use

And in that case you probably trust MIT more than the other networks you might be on

So it might be worth it, but it's something for you to think about

(student) When you say you could encrypt it with, what was it, DNS, how would you do that?

(Jon) So, DNS is the way that people turn domain names,

Or your computer turns domain names into IP addresses to know what computer to connect to

And that protocol by default is in plain text. There's nothing encrypted about it

There are various ways to encrypt your DNS traffic

Some of them are standardized and some of them are not

I won't go into the exact mechanics here, but you should google it and look at some of the ways

Okay. The last thing I want to talk about is Markdown

So there is a high chance that some of you are going to write text over the remaining part of your life

And you will want to mark up that text in various simple ways

And one thing you could do is start up Word or use LaTeX or something like that to mark up your documents

But that is a pretty heavy-handed approach

Instead it would be nice if we could just sort of

Write things the way we feel like they should be

I don't know how to describe it in a better way but sort of the natural way where if you want something,

If you want to put emphasis on a word you just put like stars around it or something, and then it just works

Markdown is essentially that. It is a way to try to encode the way that we often write text somewhat naturally

Into a markup language that lets you write things like bold text, links, lists, that sort of stuff

In fact, all of the lecture notes for this class have been written using Markdown

And Markdown is really very straightforward. The basic rules are in the notes

But the basic things you need to know is, in Markdown if you put stars

Around a word, that word is emphasized. Or some sequence of words. If you put double stars

That word is emphasized strongly, also known as bold

There are various other things you could do, like if you put a dash before a line it is now a list

And there's one list item and you can amend list items

If you put "1." in front, or some other number, it becomes a numbered list

If you put a pound sign in front of something it becomes a header, like some kind of title header

If you put multiple of them, they become subheadings and you can keep adding more to these

If you want to write code you can put a single backtick, followed by some code, followed by a backtick

And now that is rendered in Monospaced Font

If you want multiple lines of code, you do a triple backtick, and then code,

And then maybe some more code and then triple backtick

And in many cases like if you're on GitHub for example,

You can even type the name of a language up here after the backticks, without a space

And it will be syntax highlighted in the language of your choice

This is a really handy thing that is supported in so many websites nowadays you might not even realize

Like in Facebook Messenger you can use many of these

They don't actually officially say they support Markdown anywhere

But many of these things just like sort of happen to work and it's worth learning

At least the basics and just start using them. You can do links and stuff as well, but that's already in the notes

Any questions about Markdown?

Right, Anish, you're up

(Anish) Is my microphone working?

Is this working? Can you guys hear me in the back?

The light's green. Oh, I think I can hear it. Okay, great

So, continuing with our theme of random topics that are all unrelated to the previous topics we've been talking about,

The next thing we're going to talk about is a program called "Hammerspoon"

Which is a tool for doing desktop automation on Mac OS

And I think there's similar tools for Windows and Linux, a lot of the ideas can carry over

You can google it if you want to figure out how to do these things on other platforms

But basically Hammerspoon is a program that lets you write Lua scripts, scripts in a programming language

That interact with various operating system functionality

So you can write code that interacts with the keyboard and mouse and connects that to window management,

To display management, the file system, battery and power management, Wi-Fi... All sorts of stuff

Like basically all the things that your operating system manages, this tool lets you hook into those things

And so it can let you do all sorts of neat things by writing just a couple lines of code

Just some examples of cool things you can do with this tool are - you can bind hotkeys to move windows to

Specific locations. So a demonstration of this is here. I have this window open. I press, in my particular setup,

"Option+Command+Right" and this window moves to the right. "Option+Command+Left", this window moves to the left

And I have a couple other shortcuts for moving things to various places

And so I can kind of have an effect similar to tiling window managers that Jon was talking about earlier

I can move windows to different parts of my screen to set things up in a particular way rather than have to use the mouse

To position things where I want them to be and then like click and drag to resize windows to the right shape

Just a keyboard shortcut can do the trick

But this tool is not limited to just moving windows around and binding that to particularly keyboard shortcuts

You can do other things like create a menu bar button with a bunch of different options

And you can bind those different options to do different things. So in my particular case

I've created this little menu and then I have a bunch of different things that I do reasonably frequently and clicking on these things

Invokes a particular Lua function that I've written that interacts with this library. So for example

Here, this "Rescue windows" thing is a particular thing where I often work with multiple displays

And sometimes my operating system gets confused and I have some window that ends up off of my display and

How do I how do I get this thing back?

Well, that's what this... Whoops, not that

That's what this "Rescue windows" thing does. It brings windows that are off the screen back onto the screen

Another neat thing I have setup here is I have particular layouts that I've named

So like a dorm, and a Lab and a Laptop layout

So for example in my Lab

I have this screen and I have another screen and have another screen besides that in a different orientation

And I have this particular setup that I want where I want

Maybe my terminal full screen on here, and my chat program over here and

This screen split up into five segments with different programs in different places

Here I can, when I show up to Lab I can just go here and click "Layout Lab" and it will invoke some code

Which is not all that complicated like 10 lines of code describes a particular layout

And it will instantiate that layout and put all the things where they need to go

I could even in theory automate some of these things where my computer could figure out like I plug in a display and my computer knows

Oh, this is the display that you have in your lab. Let me automatically instantiate this layout for you

That's another thing you can do with Hammerspoon. And also other wacky things you can do like you can do things like

It can detect your Wi-Fi network that you're on so it knows kind of where you are

Maybe I have a different Wi-Fi network name at home versus in lab and I can do things like when I show up to lab

Automatically mute my speakers. So I don't have like embarrassing music play out loud in my lab

Another kind of cool example is

So I have a Mac

It has a fancy power supply and a lot of my friends have

Computers that look the same as this and their power supply bricks look the same as mine and

Sometimes I use their power brick because I forgot mine at home or something

This tool can actually with like three or four lines of code

Do neat things like show you a warning like it'll pop up a warning if you've accidentally taken your friend's power supply and plugged it

Into your computer instead of using your own

So at a high level this tool lets you run arbitrary Lua code and do things like bind it to menu buttons or key presses and

It interacts with a large part of the operating system in order to do all sorts of cool stuff

So that is Hammerspoon, any questions about that?

Cool. Moving on to the next topic

Completely unrelated to the previous one, it's booting and live USBs

So the operating system on your computer, Windows or Mac OS

or whatever you're used to is not exactly the first thing that runs on your machine when it turns on. There's something else that

happens in the boot process before your operating system is loaded and

There's some interesting stuff that you can do there. So you might have seen when you turn on your computer

It says something like press F9 to configure the BIOS or press F12 to enter the boot menu

The particular key sequences may depend on your machine and specific configuration

But this is a general pattern and you can configure all sorts of interesting Hardware related stuff here

So it's worth checking out

And another thing you can do in this boot menu is you can have your computer start off from an alternate boot device

so by default like my laptop here has a solid-state drive and it boots Mac OS when it turns on but I can also

say plug in a USB flash drive that has an operating system installed on the flash drive and then at boot tell my computer to

Boot from that flash drive instead of the built-in solid-state disk, and this is useful

for example

If I've broken my operating system

Install and I want to do something like get the data off my computer or maybe want to fix the operating system

Like maybe there's some critical files somewhere that I've deleted or I forgot my password

I need to go like tweak some files in order to reset it booting from a live USB

Booting from the separate operating system that's installed on a flash drive can let me do that like boot up my operating system

Mount the hard disk that's on my current machine

I'm working on and then go make some tweaks or copy data off of that

And so live USBs are really useful and in the lecture notes, we've linked to a tool that can help you create them really easily

Any questions about the boot process or live USBs?

All right, next topic is virtual machines, Vagrant, Docker, the cloud and OpenStack

I think last year we had an entire lecture on this topic this year

We're going to condense it into like one minute

So at a high level virtual machines and similar tools like containers let you emulate a whole computer system within your current machine

so like I'm running Mac OS here, but within my Mac OS

environment I can

Simulate a machine that's running say Ubuntu or some other operating system

And this is a nice way of creating an isolated environment for testing or for development or for exploration

For example doing things like running potentially malicious code that should be isolated from my current environment. I

Think the most common use case for programmers is to use virtual machines are containers to create development environments

So I'm using Mac OS and I have some of services and stuff and libraries installed in my current machine

But I might want for example

I'm working on some web programming project and I want it to run on an

Debian machine and I need

Postgres, like a database server installed rather than install that all on my Mac OS machine

I can instantiate this new machine just for the development purposes

Now virtual machines, like that's a general concept. There are a bunch of programs that let you

that are called virtual machine hypervisors that

Support this functionality on your machine and then there are tools that let you script these hypervisors in order to specify machine

Configurations like operating system and like what packages you want installed and what services you want installed in plain text

And so this is an example on the screen right here. And this is done using a system called Vagrant.

It's linked in the lecture notes. You can look into this if you're curious.

So basically and the short plain text file I can specify okay. I want a machine that's running Debian

It should have "postgres" and Redis and Python and stuff installed on it.

And then, once I have this configuration,

I can just type in "vagrant up", and what it does is, it reads this file and instantiates a new machine based on this

configuration. And then, after I've done that I can do "vagrant ssh" to SSH into this virtual machine.

So it's not a remote machine running on some other piece of hardware somewhere,

It's just simulated on my own machine, but now here I have an Ubuntu box like I do now, let's be really stashe

It's like or sorry not a bunch of debian here with all the things

I want installed in here and I can do my development inside this isolated environment,

and not, kind of, install all this junk on my MacOS machine.

Now so that's vagrant there's similar tools like docker that are conceptually similar but use containers instead of virtual machines

It's a distinction that we're not going to talk about in too much detail right now

And so you can run VMs on your own computer

but you can also rent virtual machines on the cloud and so

It's a nice way to get instant access to like one example is you might want a computer

That's always on always connected to the internet and has a public IP address. Like maybe you want to run a web server

That's always available or you want to run some other service like say a slack bot or something like that

Well a virtual machine

rented on the cloud is one nice way to get that and these are pretty cheap for like a low capacity machine with a

small CP and small amount of disk space and

You might want to do is get access to a machine that's really powerful like with a lot of CPU cores or with a lot

Of RAM or with a whole bunch of GPUs for some specific purpose like say you're doing deep learning or so

You're doing some other sorts of sensitive computation

Well, that's another thing you can do with VMs on the cloud

And finally, you can get access to many more machines than you have physical access to. Like if I need a thousand machines

But only for two minutes to do some very parallel tasks. That's something I can easily do with virtual machines and

Popular services for doing this are things like Amazon AWS or Google cloud

And if you're a member of MIT CSAIL, you can also get free VMs for research purposes using the CSAIL OpenStack

And so this is also linked in the lecture notes

So any questions about virtual machines, or Vagrant, Docker, or anything like that?

So the question is when I say I'm running Ubuntu, or, actually, in this case, it's Debian

when I'm running Ubuntu here, do I have, like, Ubuntu installed on my machine, or

What exactly is going on here? So, basically, what Vagrant did for me when I type "vagrant up" is,

because I've specified I want Debian here, it downloaded Debian from the internet, like, set up a disk image for this new machine,

installed Debian into that disk image, then went to install these programs

So, like, yes, this is on my computer. But, all of this is just in a particular file that's a disk image.

And then, I'm emulating a machine that is basically completely isolated from my current machine.

This is being run as a process on my current machine. Does that answer the question? Any other questions about VMs?

Great.

Next topic is also going to be a quick mention -

So,

a lot of you are programmers, and you're used to writing programs

in a tool like Vim, or some other editor that you're comfortable with.

Another thing that can be really neat to use for particular tasks, is something called a notebook programming environment.

And this is a more interactive way of writing programs. Here on the screen, I have a demo.

This is something called Jupiter notebook and it can be used for writing Python programs. I think they also support some other languages.

And, basically, this is a nice way of doing interactive programming.

So, normally, you're used to writing a big program in a file, or a collection of files,

and once you're done writing it, you can just run the whole program. But, this lets you be a little bit more flexible and

run little snippets of code at a time. Like, for example, I can break my program into these little pieces.

It's just some random code I wrote. And, I can say, "execute this cell", and I press a particular key combination to execute the cell.

But then, I can go back, and tweak my program a little bit. Like, say I want to have this be lowercase instead.

Then, I can execute this cell, and then go and evaluate this thing, and

this way I can, kind of, run little snippets of code, within a Python environment.

And, it's a nice way of building up programs, piece by piece, rather than having to write everything at once.

This is really useful for particular research purposes, like I think a lot of people use these for doing machine learning work, for example.

Any questions about the idea of notebook programming environments? They're worth checking out.

Oh, so the question is, "This looks like it's online, is there an offline version of Jupiter notebooks?"

So, actually, this is the thing that runs in the browser, but it's running locally...

So, I don't know if you can see it on the screen, because it's kind of small, but up here

it says, "localhost:8888".

Here, I have running on my own local machine, a Jupiter notebook. And, they've just built it

so it runs within the web browser.

That being said, there are also online

Jupiter notebooks that you can use, where the Python kernel is actually running on some remote machine. You might want to do this

for example, like on my laptop,

I don't have a fancy GPU, but in my room, I have a machine with a fancy GPU. And so, when I'm doing machine learning

work, I often SSH into that machine, run a Jupiter notebook on there, and then open up the interface, in my local web browser,

so I have access to that powerful GPU running on my different machine.

Any other questions?

Great.

The final thing we're going to talk about today is

Github. So, we touched on this a little bit during the version control lecture.

But, Github is one of the most popular platforms for open-source software development.

It hosts source code. It hosts git repositories. But, they also have other tools for managing a project.

And, like, a lot of the tools we've talked about in this class are hosted on Github. For example,

like, Hammerspoon, the thing we just talked about, is developed on Github

It's really easy to get started

contributing to open-source projects on Github to help improve the tools that you use every day.

There are two primary ways you can contribute to projects on Github.

Let's open up some repository. We can actually go to the Github repository for the class website.

So, this is an open-source software project.

And...

Let's zoom in a little bit. So the two ways you can contribute to projects on Github -

The two main ways are through issues and pull requests

One thing that's actually really helpful to developers, and also pretty lightweight and easy for users to do, is to report issues with a

software project. Like, say you're using somebody's program and you encounter some bug...

Writing a high quality issue is actually super helpful to developers and hopefully doesn't take you too much time.

And so, you can go to here, like find the project on GitHub, go to the issues page, and click on new issue, and then

write some high quality bug report. And then, hopefully the developer will respond and fix the issue for you.

So, for example, for this class,

like, one of the students in this class pointed out an issue with our lecture notes, and after she pointed it out,

I said, okay, like, that looks like a reasonable thing. Let's fix it. And in this particular case, instead of fixing it myself,

I actually asked this person, "Do they just want to fix it for me?" And so, that leads into the other thing

I want to talk about: issues and pull requests. So, pull requests are the second way to contribute to projects on Github. And,

this involves actually contributing code back to the project. And

so if we look at the pull request for this particular project, you'll see that a bunch of people have submitted code changes.

And,

the process for doing so - so this is showing the difference, the patch, that this person submitted.

Basically, the process for creating pull requests is a little bit more involved than submitting issues. Like you're not just submitting text:

you're actually going to modify their source code.

And so, we've linked to some guides that explain the process in a little bit more detail.

But, at a high level, what you do is, you take the repository on Github, "fork" it, and then download it locally,

so now have your own, local copy.

Then, you can go and work on it, do some development work, and fix a bug or add a feature, and then, eventually, you

send what's called a "pull request" back to the original developer. So, you say, 'Here. I've made some changes.

Can you please incorporate them back into the original project?'

And, after that point, what usually happens with these projects, is that the maintainer will go back and forth with you, giving you feedback on

the changes you've proposed. And, eventually, once everybody's happy, they will "merge" in your changes, and they'll be available to everybody who uses the project.

So, that is how you can contribute to projects on GitHub and make software better for everybody. And, so any questions about GitHub?

Cool, okay. So, that is it for the topics for today. Any questions about the lecture overall?

Great, okay, so before -

before we finish, a quick description about tomorrow's lecture: so, today was all the topics we thought are interesting,

we should talk about,

tomorrow's lecture is gonna be about all the topics you think are interesting and that we should talk about. So, tomorrow is going to be

a Q&A lecture. And, after today,

uh, after the lecture, we'll submit,

we'll email out a link where you can submit questions for us to answer. And so, please go and fill that out,

otherwise, we won't have too much to talk about tomorrow.

Great, so hopefully, see you tomorrow in our Q&A lecture.