Lecture 4: Data Wrangling (2020)

all right so welcome to today's lecture

which is going to be on data wrangling

and data wrangling might be a phrase it

sounds a little bit odd to you but the

basic idea of data wrangling is that you

have data in one format and you want it

in some different format and this

happens all of the time I'm not just

talking about like converting images but

it could be like you have a text file or

a log file and what you really want this

data in some other format like you want

a graph or you want statistics over the

data anything that goes from one piece

of data to another representation of

that data is what I would call data

wrangling we've seen some examples of

this kind of data wrangling already

previously in the semester like

basically whenever you use the pipe

operator that lets you sort of take

output from one program and feed it

through another program you are doing

data wrangling in one way or another but

we're going to do in this lecture is

take a look at some of the fancier ways

you can do data wrangling and some of

the really useful ways you can do data

wrangling in order to do any kind of

data wrangling though you need a data

source you need some data to operate on

in the first place and there are a lot

of good candidates for that kind of data

we give some examples in the exercise

section for today's lecture notes in

this particular one though I'm going to

be using a system log so I have a server

that's running somewhere the Netherlands

because that seemed like a reasonable

thing at the time and on that server

it's running sort of a regular logging

daemon that comes with system Deeb's

it's a sort of relatively standard Linux

logging mechanism and there's a command

called journal CTL on Linux systems that

will let you view the system log and so

what I'm gonna do is I'm gonna do some

transformations over that log and see if

we can extract something interesting

from it you'll see though that if I run

this command I end up with a lot of data

because this is a log that has just like

there's a lot of stuff in it right a lot

of things have happened on my server and

this goes back to like January first and

their logs that go even further back on

this there's a lot of stuff so the first

thing we're gonna do is try to limit it

down to you only

one piece of content and here the grep

command is your friend so we're gonna

pipe this through grep and we're gonna

pipe for SSH right so SSH we haven't

really talked to you about yet but it is

a way to access computers remotely

through the command line and in

particular what happens when you put a

server on the public Internet is that

lots and lots of people around the world

to try to connect to it and log in and

take over your server and so I want to

see how those people are trying to do

that and so I'm going to grep for SSH

and you'll see pretty quickly that this

also generates a bunch of content at

least in theory this is gonna be real

slow there we go so this generates tons

and tons and tons of content and it's

really hard to even just visualize

what's going on here so let's look at

only what user names people have used to

try to log into my server so you'll see

some of these lines say disconnected

disconnected from invalid user and then

some user name I want only those lines

that's all I really care about I'm gonna

make one more change here though which

is if you think about how this pipeline

does if I here do this connected from so

this pipeline at the bottom here what

that will do is it will send the entire

log file over the network to my machine

and then locally run grep to find only

the lines to contained ssh and then

locally filter them further this seems a

little bit wasteful because i don't care

about most of these lines and the remote

site is also running a shell so what I

can actually do is I can have that

entire command run on the server right

so I'm telling you SSH the command I

want you to run on the server is this

pipeline of three things and then what I

get back I want to pipe through less so

what does this do well it's gonna do

that same filtering that we did but it's

gonna do it on the server side and the

server is only going to send me those

lines that I care about and then when I

pipe it locally through the program

called less less is a pager you'll see

some examples of this you've actually

seen some of them already like when you

type man and some command that opens in

a pager and a pagers is a convenient way

to take a long piece of content and fit

it into your term

window and have you scrolled down and

scroll up and navigate it so that it

doesn't just like scroll past your

screen and so if I run this it still

takes a little while because it has to

parse through a lot of log files and in

particular grep is buffering and

therefore it decides to be relatively

unhelpful

I may do this without let's see if

that's more helpful why doesn't it want

to be helpful to me fine I'm gonna cheat

a little just ignore me

or the internet is really slow those are

two possible options luckily there's a

fix for that because previously I have

run the following command so this

command just takes the output of that

command and sticks it into a file

locally on my computer alright so I ran

this when I was up in my office and so

what this did is it downloaded all of

the SSH log entries that matched

disconnect from so I have those locally

and this is really handy right there's

no reason for me to stream the full log

every single time because I know that

that starting pattern is what I'm going

to want anyway so we can take a look at

SSH dot log and you will see there are

lots and lots and lots of lines that all

say disconnected from invalid user

authenticating users etc right so these

are the lines that we have to work on

and this also means that going forward

we don't have to go through this whole

SSH process we can just cat that file

and then operate it on it directly so

here I can also demonstrate this pager

so if I do cat s is a cat SSH dot log

and I pipe it through less it gives me a

pager where I can scroll up and down

make that a little bit smaller maybe so

I can scroll this file screw through

this file and I can do so with what are

roughly vim bindings so control you to

scroll up control D to scroll down and

cue to exit this is still a lot of

content though and these lines contain a

bunch of garbage that I'm not really

interested in what I really want to see

is what are what are these user names

and here the tool that we're going to

start using is one called sent said is a

stream editor that's modify or it's it's

a modification of a much earlier program

called edie which was a really weird

editor that none of you will probably

want to use yeah Oh tsp is the name of

my the remote computer I'm connecting to

so said is a stream editor and it

basically lets you make changes to the

contents of a stream you can think of it

a little bit like doing replacements but

it's actually a full programming

language

over the stream that is given one of the

most common things you do with said

though is to just run replacement

expressions on an input stream what do

these looks like well let me show you

here I'm gonna pipe this sue said and

I'm going to say that I want to remove

everything that comes before

disconnected from so this might look a

little weird the observation is that the

date and the host name and the sort of

process ID of the SSH daemon I don't

care about I can just remove that

straightaway and I can also remove that

like disconnected from bit because that

seems to be present in every single log

entry so I just want to get rid of it

and so what I write is a set expression

in this particular case it's an S

expression which is a substitute

expression it takes two arguments that

are basically enclosed in these slashes

so the first one is the search string

and the second one which is currently

empty is a replacement string so here

I'm saying search for the following

pattern and replace it with blank and

then I'm gonna pipe it into less at the

end do you see that now what it's done

is trim off the beginning of all these

lines and that seems really handy but

you might wonder what is this pattern

that I've built up here right this is

this dot star what does that mean this

is an example of a regular expression

and regular expressions are something

that you may have come across in

programming in the past

but it's something that once you go into

the command line you will find yourself

using a lot especially for this kind of

data wrangling regular expressions are

essentially a powerful way to match text

you can use it for other things than

text too but Texas the most common

example and in regular expressions you

have a number of special characters that

say don't just match this character but

match for example a particular type of

character or a particular set of options

it essentially generates a program for

you that searches the given text dot for

example means any single

character and star if you follow a

character with a star it means zero or

more of that character and so in this

case is pattern of saying zero or more

of any character followed by the literal

string disconnected from I'm saying

match that and then replace it with

blank regular expressions have a number

of these kind of special characters that

have various meanings you can take

advantage of I talked about star which

is zero or more

there's also Plus which is one or more

right so this is saying I want the

previous expression to match at least

once

you also have square brackets so square

brackets let you match one of many

different characters so here let us

build up a string list something like a

BA and I want to substitute a and B with

nothing okay so here what I'm telling

the pattern to do is to replace any

character that is either A or B with

nothing so if I make the first character

B it will still produce BA you might

wonder though why did it only replace

once well it's because what regular

expressions will do especially in this

default mode is they will just match the

pattern once and then apply the

replacement once per line that is what's

said normally does you can provide the G

modifier which says do this as many

times as it keeps matching which in this

case would erase the entire line because

every single character is either an A or

a B if I added a C here and remove

everything but the C if I added other

characters in the middle of this string

somewhere they would all be preserved

but anything that is an A or and B is

removed you can also do things like add

modifiers to this for example

what would this do this is saying I want

zero or more of the string a B and I'm

gonna replace them with nothing

this means that if I have a standalone a

it will not be replaced if I have a

standalone B it will not be replaced but

if I have the string a B it will be

removed which yeah what are they said is

stupid

the - a here is because said is a really

old tool and so it supports only a very

old version of very cool expressions

generally you will want to run it with -

capital e which makes it use a more

modern syntax that supports more things

if you are in a place where you can't

you have to prefix these with back

slashes to say I want the special

meaning of parenthesis otherwise they

were just match a literal parenthesis

which is probably not what you want so

notice how this replaced the a B here

and it replaced the a be here but it

left this C and it also left the a at

the end because that a does not match

this pattern anymore and you can group

these patterns in whatever ways you want

you also have things like alternations

you can say anything that matches a b or

b c i want to remove and here you'll

notice that this a b got removed this bc

did not get removed even though it

matches the pattern because the a b had

already been removed this a b is removed

right but the c stays in place this a b

is removed and this c states because it

still does not match that if I made this

if I remove this a then now this a B

pattern will not match this B so it'll

be preserved and then BC will match BC

and it'll go away

Regulus presence can be all sorts of

complicated when you first encounter

them and even once you get more

experience with them they can be

daunting to look at and this is why very

often you want to use something like a

regular expression debugger which we'll

look at in a little bit but first let's

try to make up a

pattern that will match the logs and and

match the logs that we've been working

with so far so here I'm gonna just sort

of extract a couple of lines from this

file let's say the first five so these

lines all now look like this right and

what we want to do is we want to only

have the user name okay so what might

this look like well here's one thing we

could try to do actually let me show you

one except one thing first let me take a

line that says something like

disconnected from invalid user

disconnected from maybe four to one one

whatever okay so this is an example of a

login line where someone tried to login

with the username disconnected from

missing an S disconnected thank you

you'll notice that this actually removed

the username as well and this is because

when you use dot star and any of these

sort of range expressions indirect

expressions they are greedy they will

match as much as they can so in this

case this was the username that we

wanted to retain but this pattern

actually matched all the way up until

the second occurrence of it or the last

occurrence of it and so everything

before it including the username itself

got removed and so we need to come up

with a slightly clever or matching

strategy than just saying sort of dot

star because it means that if we have

particularly adversarial input we might

end up with something that we didn't

expect okay so let's see how we might

try to match these lines let's just do a

head first well let's try to construct

this up from the beginning we first of

all know that we want - capital e right

because we want to not have to put all

these back slashes everywhere

these lines look like they say from and

then some of them say invalid but some

of them do not right this line has

invalid that one does not question mark

here is saying zero or one so I want

zero or zero or one of invalid space

user what else well that's going to be a

double space so we can't have that and

then there's gonna be some username and

then there's gonna be what exactly is

gonna be what looks like an IP address

so here we can use our range syntax and

say zero to nine and a dot right that's

what IP addresses are and we want many

of those then it says porch so we're

just going to match a literal port and

then another number zero to nine and

we're going to wand plus of that the

other thing we're going to do here is

we're going to do what's known as

anchoring the regular expression so

there are two special characters and

regular expressions there's carrot or

hat which matches the beginning of a

line and there's dollar which matches

the end of a line so here we're gonna

say that this regression has to match

the complete line the reason we do this

is because imagine that someone made

their username the entire log string

then now if you try to match this

pattern it would match the username

itself which is not what we want

generally you will want to try to anchor

your patterns wherever you can to avoid

those kind of oddities okay let's see

what that gave us that removed many of

the lines but not all of them so this

one for example includes this pre off at

the end so we'll want to cut that off if

there's a space pre off square brackets

our specials we need to escape them

right now let's see what happens if we

try more lines of this no it still gets

something weird some of these lines are

not empty right which means that the

pattern did not match this one for

example it says authenticating user

instead of invalid

user okay so as to match invalid or

authenticated zero or one time before

user how about now okay that looks

pretty promising but this output is not

particularly helpful right here we've

just erased every line of our log files

successfully which is not very helpful

instead what we really wanted to do is

when we match the username right over

here we really wanted to remember what

that username was because that is what

we want to print out and the way we can

do that in regular expressions is using

something like capture groups so capture

groups are a way to say that I want to

remember this value and reuse it later

and in regular expressions any bracketed

expression any parenthesis expression is

going to be such a capture group so we

already actually have one here which is

this first group and now we're creating

a second one here notice that these

parentheses don't do anything to the

matching right because they're just

saying this expression as a unit but we

don't have any modifiers after it so

it's just match one-time and then the

reason matching groups are are useful or

capture groups are useful is because you

can refer back to them in the

replacement so in the replacement here I

can say backslash two this is the way

that you refer to the name of a capture

group in this say I'm in this case I'm

saying match the entire line and then in

the replacement put in the value you

captured in the second capture group

right remember this is the first capture

group and this is the second one and

this gives me all the usernames now if

you look back at what we wrote this is

pretty complicated right it might make

sense now that we walk through it and

why it had to be the way it was but this

is like not obvious that this is how

these lines work and this is where a

regular expression debugger can come in

really really handy so we have one here

there are many online but here I've sort

of pre filled in this expression that we

just used and notice that it it tells me

all the matching does in fact now this

window is a little small with this font

size but if I do hear this explanation

says dot star matches any character

between zero and unlimited times

followed by disconnected from literally

followed by a capture group and then

walks you through all the stuff and

that's one thing but it also lets you've

given a test string and then matches the

pattern against every single test string

that you give and highlights what the

different capture groups for example are

so here we made user a capture group

right so it'll say okay the full string

matched right the whole thing is blue so

it matched Green is the first capture

group red is the second capture group

and this is the third because preauth

was also put into parenthesis and this

can be a handy way to try to debug your

regular expressions for example if I put

disconnected from and let's add a new

line here and I make the username

disconnected from now that line already

had the username be disconnect from

great here me of thinking ahead you'll

notice that with this pattern this was

no longer a problem because it got

matched the username what happens if we

take this entire line or this entire

line and make that the username now what

happens it gets really confused right so

this is where regular expressions can be

a pain to get right because it now tries

to match it matches the first place

where username appears or the first

invalid in this case the second invalid

because this is greedy we can make this

non greedy by putting a question mark

here so if you suffix a plus or a star

with a question mark it becomes a non

greedy match so it will not try to match

as much as possible and then you see

that this actually gets parsed correctly

because this dots

we'll stop at the first disconnected

from which is the one that's actually

emitted by SSH the one that actually

appears in our logs as you can probably

tell from the explanation of this so far

regular expressions can get really

complicated and there are all sorts of

weird modifiers that you might have to

apply in your pattern the only way to

really learn them is to start with

simple ones and then build them up until

they match what you need often you're

just doing some like one-off job like

when we're hacking out the user names

here and you don't need to care about

all the special conditions right you

don't have to care about someone having

the SSH username perfectly match your

login format that's probably not

something that matters because you're

just trying to find the usernames but

regular expressions are really powerful

and you want to be careful if you're

doing something where it actually

matters you had a question

regular expressions by default only

match per line anyway they will not

match across new lines so so the way

that said works is that it operates per

line and so said we'll do this

expression for every line okay questions

about regular sessions or this pattern

so far it is a complicated pattern so if

it if it feels confusing like don't be

worried about it look at it in the

debugger later yep so so keep in mind

that the we're assuming here that the

user only has control over their

username right so the worst that they

could do is take like this entire entry

and make that the username let's see

what happens right so that's the works

and the reason for this is this question

mark means that the moment we hit the

disconnect keyword we start parsing the

rest of the pattern right and the

first occurrence of disconnected is

printed by SSH before anything the user

controls so in this particular instance

even this will not confuse the pattern

yep if well so if you're writing a this

sort of odd matching will in general

when you're doing data wrangling is like

not security it's not security related

but it might mean that you get really

weird data back and so if you're doing

something like plotting data you might

drop data points that matter you might

parse out the wrong number and then like

your plot suddenly have data points that

weren't in the original data and so it's

more that if you find yourself writing a

complicated regular expression like

double check that it's actually matching

what you think it's matching and even if

it's not security related and as you can

imagine these patterns can get really

complicated like for example there's a

big debate about how do you match an

email address with a regular expression

and you might think of something like

this so this is a very straightforward

one that just says letters and numbers

and rotor scores some percent followed

by a plus because in Gmail you can have

pluses in email addresses with a suffix

in this case the plus is just for any

number of these but at least one because

you can't have an email address that

doesn't have anything before the ad and

then similarly after the domain right

and the top-level domain has to be at

least two characters and can't include

digits right you can have it calm but

you can't have adopt seven it turns out

this is not really correct right there

are a bunch of valid email addresses

that will not be matched by this and

they're a bunch of invalid email

addresses that will be matched by this

so there are many many suggestions and

there are people who've built like full

test suites to try to see which regular

expression is best and this is this

particular one is for URLs there are

similar ones for email where they found

that the best one is this one I don't

recommend you trying to understand this

pattern but this one apparently will all

most perfectly match the what the like

internet standard for email addresses

says as a valid email address and that

includes all sorts of weird Unicode code

points this is just to say regular

expressions can be really hairy and if

you end up somewhere like this there's

probably a better way to do it for

example if you find yourself trying to

parse HTML or something or parse like

parse JSON where they're expressions you

should probably use a different tool and

there is an exercise that has you do

this not with the regular sessions point

you yeah that it's there's all sorts of

suggestions and they give you deep deep

dives into how they works if you want to

look that up it's it's in the lecture

notes okay so now we have the sister of

user names so let's go back to data

wrangling right like this list of user

names is still not that interesting to

me right let's let's see how many lines

there are so if I do WC - oh there are

one hundred and ninety eight thousand

lines so WC is the word count program -

L makes it count the number of lines

this is a lot of lines then if I start

scrolling through them that still

doesn't really help me right like I need

statistics over this I need aggregates

of some kind and the send tool is like

useful for many things it gives you a

full programming language it can do

weird things like insert text or only

print matching lines but it's not

necessarily the perfect tool for

everything right like sometimes there

are better tools like for example you

could write a line counter instead you

just should never said it's a terrible

programming language except for

searching and replacing but there are

other useful tools so for example

there's a tool called sort so sort this

is also not going to be very helpful but

sort takes a bunch of lines of input

sorts them and then prints them to your

output so in this case I now get the

sorted output of that list it is still

two hundred thousand lines long so it's

still not very helpful to me but now I

can combine it

the tool called unique so unique we'll

look at a sorted list of lines and it

will only print those that are unique so

if you have multiple instances of any

given line it will only print it once

and then I can say unique - C so this is

gonna say count the number of duplicates

for any lines that are duplicated and

eliminate them what does this look like

well if I run it it's gonna take a while

there were thirteen zze user names there

were ten ZX VF user names etc there and

I can scroll through this this is still

a very long list right but at least now

it's a little bit more collated than it

was let's see how many lines I'm dumped

in now okay

twenty-four thousand lines it's still

too much it's not useful information to

me but I can keep burning down this with

more tools for example what I might care

about is which user names have been used

the most well I can do sort again and I

can say I want a numeric sort on the

first column of the input so - n says

numeric sort - K lets you select a white

space separated column from the input to

sort my and the reason I'm giving one

comma one here is because I want to

start at the first column and stop at

the first column alternatively I could

say I want you to sort by this list of

columns but in this case I just want to

sort by that column and then I want only

the ten last lines so sort by default

will output in ascending order so the

the ones with the highest counts are

gonna be at the bottom and then I want

only lost ten lines and now when I run

this I actually get a useful bit of data

right it tells me there were eleven

thousand login attempts with the

username root there were four thousand

with one two three four five six isn't

username etc and this is pretty handy

right and now suddenly this giant log

file actually produces useful

information for me this is what I really

from that log file now maybe I want to

just like do a quick disabling of root

for example for SSH login on my machine

which I recommend you will do by the way

in this particular case we don't

actually need the k4 sort because sort

by default will sort by the entire line

and the number happens to come first but

it's useful to know about these

additional flags and you might wonder

well how would I know that these flags

exist how would I know that these

programs even exist

well the programs usually pick up just

from being told about them in classes

like here the flags are usually like I

want to sort by something that is not

the full line your first instinct should

be to type man sort and then read

through the page and then very quickly

will tell you here's how to select a

pretty good column here's how to sort by

a number etc okay what if now that I

have this like top let's say top 20 list

let's say I don't actually care about

the counts I just want like a comma

separated list of the user names because

I'm gonna like send it to myself by

email every day or something like that

like these are the top 20 usernames well

I can do this

ok that's a lot more weird commands but

their commands that are useful to know

about so awk is a column based stream

processor so we talked about said which

is a stream editor so it tries to edit

text primarily in the inputs awk on the

other hand also lets you edit text it is

still a full programming language but

it's more focused on columnar data so in

this case awk by default will parse its

input in white space separated columns

and then that you operate on those

columns separately in this case I'm

saying just print the second column

which is the user name right paste is a

command that takes a bunch of lines and

paste them together into a single line

that's the - s with the delimiter comma

so in this case for on this I want to

get a comma separated list of the top

user names which I can then do whatever

useful thing I might want maybe I want

to stick this in a config file of

disallowed usernames or something along

those lines

um awk is worth talking a little bit

more about because it turns out to be a

really powerful language for this kind

of data wrangling we mentioned briefly

what this print dollar 2 does but it

turns out the for awk you can do some

really really fancy things so for

example let's go back to here where we

just have the usernames I say let's

still do sort and unique because we

don't otherwise the list gets far too

long

and let's say that I only want to print

the usernames that match a particular

pattern let's say for example that I

want to see I want all of the usernames

that only appear once and that start

with a C and end with an e there's a

really weird thing to look for but in

all this is really simple to express I

can say I want the first column to be 1

and I want the second column to match

the following regular expression

hey this could probably just be dot and

then I want to print the whole line so

unless I mess something up this will

give me all the usernames that start

with a C end with an e and only appear

once in my log now that might not be a

very useful thing to do with the data

what I'm trying to do in this lecture is

show you the kind of tools that are

available and in this particular case

this pattern is like not that

complicated even though what we're doing

is sort of weird and this is because

very often on Linux with Linux tools in

particular and command-line tools in

general the tools are built to be based

on lines of input and lines of output

and very often those lines are going to

be have multiple columns and awk is

great for operating over columns now awk

is is not just able to do things like

match per line but it lets you do things

like let's say I want the number of

these right I want to know how many user

names match this pattern well I can do

WCHL that works just fine all right

there are 31 such user names but awk is

a programming language this is something

that you will probably never end up

doing yourself but it's important to

know that you can every now and again it

is actually useful to know about these

this might be hard to read on my screen

I just realized let me try to fix that

in a second

let's do yeah apparently fish does not

want me to do that um so here begin is a

special pattern that only matches the

zeroth line end is a special pattern

that only matches after the last line

and then this is gonna be a normal

pattern that's matched against every

line so what I'm saying here is on the

zeroth line set the variable rose to

zero on every line that matches this

pattern increment rose and after you

have matched the last line print the

value of rose and this will have the

same effect as running WCHL but all

within awk his particular instance like

WCHL is just fine but sometimes you want

to do things like you want to might want

to keep a dictionary or a map of some

kind you might want to compute

statistics you might want to do things

like I want the second match of this

pattern so you need a stateful matcher

like ignore the first match but then

print everything following the second

match and for that this kind of simple

programming in all can be useful to know

about in fact we could in this pattern

get rid of said and sort and unique and

grep that we originally used to produce

this file and do it all in awk

but you probably don't want to do that

it would be probably too painful to be

worth it it's worth talking a little bit

about the other kinds of tools that you

might want to use on the command line

the first of these is a really handy

program called BC so BC is the Berkeley

calculator I believe man BC I think BC

is originally from Berkeley calculator

anyway it is a very simple command-line

calculator but instead of giving you a

prompt it reads from standard in so I

can do something like echo 1 plus 2 and

pipe it to BC - shell because many of

these programs normally operate in like

a stupid mode where they're unhelpful so

here it prints 3 Wow very impressive but

it turns out this can be really handy

imagine you have a file with a bunch of

lines

let's say something like oh I don't know

this file and let's say I want to sum up

the number of logins the number of user

names that have not been used only once

all right so the ones where the count is

not equal to one I want to print just

the count right this is me give me the

counts for all the non single-use user

names and then I want to know how many

are there of these notice that I can't

just count the lines that wouldn't work

right because there are numbers on each

ran I want to sum well I can use paste

to paste by plus so this paste every

line together into a plus expression

right and this is now an arithmetic

expression so I can pipe it through BCL

and now there have been hundred and

ninety one thousand logins that share to

username with at least one other login

again probably not something you really

care about but this is just to show you

that you can extract this data pretty

easily and there's all sort of other

stuff you can do with this for example

there are tools so that you compute

statistics over inputs so for example

for this list of numbers that's that I

just took the numbers and just print it

out just the distribution of numbers I

could do things like use our our is the

separate programming language that's

specifically built for a statistical

analysis and I can say let's see if I

got this right

this is again a different programming

language that you would have to learn

but if you already know R or you can

pipe them through all their languages

too like so so this gives me summary

statistics over that input stream of

numbers so the median number of login

attempts per user name is 3 the max is

10,000 that was route

we saw before I'll tell me the average

was 8 for this might not matter in this

particular instance like this might not

be interesting numbers but if you're

looking at things like output from your

benchmarking script or something else

where you have some numerical

distribution and you want to look at

them these tools are really handy we can

even do some simple plotting if we

wanted to right so this has a bunch of

numbers let's do let's go back to our

sort and k-11 and look at only the two

top 5 new plot is a plotter that lets

you take things from standard in I'm not

expecting you to know all of these

programming languages because they

really are programming languages in

their own right but is it just show you

what is possible right so this is now a

histogram of how many times each of the

top 5 user names have been used for my

server since January 1st and it's just

one command line it's somewhat

complicated command line but it's just

one command line thing that you can do

there are two sort of special types of

data wrangling that I want to talk to

you about in the in the last little bit

of time that we have and the first one

is command line argument wrangling

sometimes you might have something that

actually we looked at in the last

lecture like you have things like find

that produces a list of files or maybe

something that produces a list of

arguments for your benchmarking script

like you want to run it with a

particular distribution of arguments

like let's say you had a script that

printed the number of iterations to run

a particular project and you wanted like

an exponential distribution or something

and this prints the number of iterations

on each line and you were to run your

benchmark for each one well here is a

tool called X args

that's your friend so X args takes lines

of input and turns them into arguments

and this is my

look a little weird see if I can come

with a good example for this so I

program in rust and rust lets you

install multiple versions of the

compiler so in this case you can see

that I have stable beta I have a couple

of earlier stable releases and I've

launched a different dated Knightley's

and this is all very well but over time

like I don't really need the nightly

version from like March of last year

anymore

I can probably delete that every now and

again and maybe I want to clean these up

a little well this is a list of lines so

I can get for nightly I can get rid of

so - V is don't match I don't want to

match to the current nightly okay so

this is al a list of dated Knightley's

maybe I want only the ones from 2019

and now I want to remove each of these

tool chains for my machine I could copy

paste each one into so there's a rust up

tool chain remove or uninstall maybe

tool chain uninstall right so I could

manually type out the name of each one

or copy/paste them but that's getting

gets annoying really quickly because I

have the list right here so instead how

about I said away this sort of this

suffix that it adds right so now it's

just that and then I use ex args so ex

args takes a list of inputs and turns

them into arguments so I want this to

become arguments to rust up tool chain

uninstall and just for my own sanity

sake I'm gonna make this echo just so

it's going to show which command it's

gonna run well it's relatively unhelpful

but are hard to read at least you see

the command it's going to execute if I

remove this echo is rust up tool chain

uninstall and then the list of

Knightley's as arguments to that program

and so if I run this it on installs

every tool chain instead of me having to

copy paste them so this is one example

where this kind of data wrangling

actually can be useful for other tasks

than just looking at data it's just

going from one

format to another you can also wrangle

binary data so a good example of this is

stuff like videos and images where you

might actually want to operate over them

in some interesting way so for example

there's a tool called ffmpeg ffmpeg is

for encoding and decoding video and to

some extent images I'm gonna set its log

level to panic because otherwise it

prints a bunch of stuff I want it to

read from dev video 0 which is my video

of my webcam video device and I wanted

to take the first frame so I just wanted

to take a picture and I wanted to take

an image rather than a single frame

video file and I wanted to print its

output so the image it captures to

standard output - is usually the way you

tell the program to use standard input

or output rather than a given file so

here it expects a file name and the file

name - means standard output in this

context and then I want to pipe that

through a parameter called convert

convert is a image manipulation program

I want to tell convert to read from

standard input and turn the image into

the color space gray and then write the

resulting image into the file - which is

standard output and I don't want to pipe

that into gzip we're just gonna compress

this image file and that's also going to

just operate on standard input standard

output and then I'm going to pipe that

to my remote server and on that I'm

going to decode that image and then I'm

gonna store a copy of that image so

remember T reads input prints it to

standard out and to a file this is gonna

make a copy of the decoded image file

ass copy about PNG and then it's gonna

continue to stream that out so now I'm

gonna bring that back into a local

stream and here I'm going to display

that in an image display err let's see

if that works

Hey right so this now did a round-trip

to my server

and then came back over pipes and

there's now a computer there's a

decompressed version of this file at

least in theory on my server let's see

if that's there a CPT's p copy PNG 2

here and CP 8 yeah hey same file ended

up on the server so our pipeline worked

again this is a sort of silly example

but let's you see the power of building

these pipelines where it doesn't have to

be textual data it's just go taking data

from any format to any other like for

example if I wanted to I can do cat dev

video 0 and then pipe that to a server

that like Anish controls and then he

could watch that video stream by piping

it into a video player on his machine if

we wanted to write it just need to know

that these thing exist there are a bunch

of exercises for this lab and some of

them rely on you having a data source

that looks a little bit like a log on

Mac OS and Linux we give you some

commands you can try to experiment with

but keep in mind that it's not it's not

that important exactly what data source

you use this is more find some data

source that where you think there might

be an interesting signal and then try to

extract something interesting from it

that is what all of the exercises are

about we will not have class on Monday

because it's MLK Day so next lecture

will be Tuesday on command line

environments any questions about what

we've guarded so far or the pipelines or

regular expressions I really recommend

that you look into regular expressions

and try to learn them they are extremely

handy both for this and in programming

in general and if you have any questions

come to office hours and we'll help you

up