Lecture 8: Metaprogramming (2020)

all right everyone let's get started

with the next lecture so today we're

gonna that tackle the topic of meta

programming and this title is a little

weird it's not entirely clear what we

mean by meta programming we couldn't

really come up with a better name for

this lecture because this lecture is

about the processes that surround the

work that you do when working with

software it is not about programming

itself necessarily but about the process

this can be things like how your system

is built how it's tested how you add

dependencies to your software that sort

of stuff that becomes really relevant

when you build larger pieces of software

but they're not really programming in

and of themselves

so the first thing we're going to talk

about in this lecture is the notion of

build systems so how many of you have

used a build system before or know what

it is okay so about 1/2 of you so for

the rest of you the the central idea

behind a build system is that you're

writing a paper you're writing software

you're like working on a class whatever

it might be and you have a bunch of

commands that you like either written

down in your shell history or you wrote

them down in a document somewhere that

you know you have to run if you want to

do a particular thing so for example

like there are a sequence of commands so

you need to run in order to build your

paper or build your thesis or just to

run the tests for whatever class you're

currently in and a build system sort of

idea is that you want to encode these

rules for what commands to run in order

to build particular targets into a tool

that can do it for you and in particular

you're going to teach this tool about

the dependencies between those different

artifacts that you might build there are

a lot of different types of tools of

this kind and many of them are built for

particular purposes particularly

languages some of them are built for

building like papers some of them are

built for building software some of them

are built for particularly programming

languages like Java or some some tools

even have built in tools for for builds

so NPM for example you might be aware if

you've done Nodejs development has a

bunch of built-in tools for doing

tracking of dependencies and building

them and building all of the dependent

stuff of your software but more

generally these are known as build

systems and at their core they all

function in a very similar way and that

is you have a number of targets these

are the things that you want to build

these are things like paper dot PDF but

they can also be more abstract things

like run the test suite or build the

binary for this program then you have a

bunch of dependencies and dependencies

are things that need to be built in

order for this thing to be built and

then you have rules that define how do

you go from a complete list of

dependencies to the given target so an

example of this might be in order to

build my paper dot PDF I need a bunch of

like plot images they're gonna go into

the paper so they need to be built but

then once they have been built how do i

construct the paper given those files so

that is what a rule is it's a sequence

of command so you run to get from one to

the other how you encode these rules

differs between different tools in this

particular class we're gonna focus on a

tool called make make is a tool that you

will find on almost any system that you

log in today like it'll be on Mac OS

it'll be on basically every Linux and

BSD system and you can pretty easily get

it on Windows um it's not great for very

complex software but it works really

well for anything that's sort of simple

to medium complexity now when you run

make to make user command you can run on

the command line and when you type make

and this is an empty directory if I type

make it just has no target specified and

no make file found stop and so it

helpfully tells you that it stopped

running but also tells you that no make

file was found make will look for a file

literally called make file in the

current directory and that is where you

encode these targets dependencies and

rules so let's try to write one let's

imagine that I'm writing this

hypothetical paper and so I'm gonna make

a make file and then this make file I'm

going to say that my paper dog PDF the

hands-on that's what the : here

indicates paper dot text was going to be

a little attack file and plot data dot

PNG and the command in order to build

this is going to be PDF latex of paper

dot Tex so for those of you who are not

familiar with this particular way of

building documents the tech is a really

handy programming language for documents

it's a really ugly language and it's a

pain to work with but it produces pretty

nice documents and the tool you use to

go from a tech file to PDF is PDF latex

and here I'm saying that I also depend

on this plot plot data PNG that's gonna

be included in my document and what I'm

really saying here is if either of those

two dependencies change I want you to

build paper PDF they both need to be

present and should they ever change I

wanted to rebuild it but I haven't

really told it how to generate this plot

data PNG so I might want to rule for

that as well

so I'm gonna define another target here

and it's gonna be it's gonna look like

this plot - % & % means and make is any

string sort of a wildcard pattern but

the cool thing is a you go and repeat

this pattern in the dependencies so I

can say that plot - % dot PNG is going

to depend on % dott data or debt that is

a common sort of suffix for data files

and it's also going to depend on some

script that's gonna actually plot this

for me and the rules for to go from one

to the other

these can be multiple lines but in my

particular case they're just one line

I'm gonna explain what this is in a

little second alright so here we're

gonna say that in order to go from a

wildcard dot dot file that matches the

wildcard in the target and a plot dot

python file run the python file with - i

which is going to be like the way we

take the mark what the input is in our

python file I'll show it to you later

dollarz star

is a special variable that is defined

for you and make file rules that matches

whatever the percentile was so if I do

plot to PNG then it's going to look for

food dot that and it dollars stars can

expand to foo so this will produce the

same file name as the one we matched

here and dollar act as a special

variable that means the name of the

target right so the output file and

hopefully what plotter py will do is

that it will take whatever the data is

here it will produce a PNG somehow and

it will write it into the file indicated

by the dollar at all right so now we

have a make file let's see what happens

if the only file in this directory is

the make file and we run make one says

no rule to make target paper dot tex

needed by paper dot PDF stop so what

it's saying here is first of all it

looked at the first rule of our file the

first target and when you give make no

arguments it tries to build whatever the

first target is this is known as the

default goal so in this case it tried to

helpfully build paper dot PDF for us and

then it looked up the dependencies and

it said well in order to build paper dot

PDF I need paper Tex and I need this PNG

file and I can't find paper dot X and I

don't have a rule for generating paper

dot X and therefore I'm gonna exit this

isn't nothing more I can do

so let's try to make some files here

let's just make like an empty paper dot

X and then type make so now it says no

rule to make target plot data dot PNG

needed by paper to PDF right so now it's

it knows that it has one dependency but

it doesn't know how to get the other one

it knows it as a target that matches but

it can't actually find its dependencies

and so it ends up doing nothing at all

it still still needs us to generate this

PNG for the input for the PNG so let's

actually put some useful stuff into

these files let's say that luckily I

have one from earlier plot da py to here

so let's so good well this text file is

this is what text looks like it's not

very pretty but

see I'm defining an empty document I'm

going to include graphics which is the

way you include a an image file I'm

going to include plot data dot PNG and

this is of course why we want a

dependency of peda paper dot PDF to be

the PNG file plot the py is also not

very interesting it just part imports a

bunch libraries it parses the - ion - Oh

arguments it loads data from the I

argument it uses library called

matplotlib which is very handy for just

quickly plotting data and it's gonna

plot the first column of the data as X's

and the second column of the data as Y's

so we're just going to have a data file

that's two columns x and y on every line

and then it saves that as a figure into

whatever the given - oh value is okay so

we need a data file that's going to be

data dot because we want plot data dot

PNG and our rules said that the way you

go from that pattern to the dot file the

dot that file is just by whatever

follows plot so if we want plot - data

then we want data dot that and then this

file we're just gonna put in some linear

coordinates because why not

that's not linear all right and now what

happens if we're gonna make well mmm

okay so what just happened well make

first ran plata py with the correct

files to generate the PNG file and then

it ran PDF latex paper dot text and all

the stuff we see below is is the output

from that tool if you wanted to we

silence the effort from this tools we

don't have to like have it mess with all

our output but in general you notice

that it ran the two commands then it

write the random perhaps unsurprisingly

in the right order and if we now do LS

in the current directory we see that we

have a bunch of files that were

generated by PDF latex but in particular

we have the PNG file which was generated

and we have the paper dot PDF and if we

open the paper - a PDF file we see that

it has one image which has the straight

line

perhaps in and of itself not a very

surprising or interesting result but

where this gets really handy is I can do

things like if I type make again make

just says paper dot PDF is up to date it

does no work whenever you run make it

tries to do the minimal amount of work

in order to produce whatever you ask it

to produce in this case none of the

dependencies have changed so there's no

reason to rebuild the paper or to

rebuild the plot if I now let's say I'm

gonna edit paper dog texts I'm gonna add

hello here an hour and make then if we

scroll up we'll see it didn't run plot

op py again because I didn't need to

none of the dependencies change but it

did run PDF latex again and indeed if we

open the paper analysis hello over there

on the other hand if I were to change

say the data file and make this point 8

an hour and make then now it plots again

because the data changed and it

regenerates the PDF because the plot

changed and indeed the paper turns out

the way we expected it to so that's not

to say that this particular pipeline is

very interesting because it's not it's

only true very very straightforward

targets and rules but this can come in

really handy when you start building

larger pieces of software or there might

be dependencies you might even imagine

that if you're writing a paper one of

your targets might be producing this

data file in the first place right so

one of the makefile

targets might be run my experiment right

run my benchmark and stick the the data

points that come out into this file and

then plot the results and then and then

and then and then all the way until you

end up with a final paper and what's

nice about this is first of all you

don't have to remember all the commands

to run you don't have to write them down

anywhere but also the tool takes care of

doing the minimal amount of work needed

often you'll find things like they'll be

too there'll be sub commands to make

like make tests right which is going to

compile your entire piece of software

and also run the tests there might be

things like make release

which builds it with optimizations

turned on and creates a tarball and

uploads that somewhere right so it's

gonna do the whole pipeline for you the

idea is to reduce the effort that you

have to do as any part of your build

process now what we saw here was a very

straightforward example of dependencies

right so we saw here that you could

declare files as dependencies but you

Kosar declare sort of transitive

dependencies right I depend on this

thing which is generated by this other

target very often when you work with

dependencies in the in the larger area

of software you'll find that your your

system ends up having many different

types of dependencies some of these are

files like we saw here some of them are

programs right like this sort of

implicitly depends on Python being

installed on my machine some of it might

be libraries right you might depend on

something like matplotlib which we

depend on here some of them might be

system libraries like open SSL or open

SSH or like low-level crypto libraries

and you don't necessarily declare all of

them very often there's sort of an an

assumption about what is installed on

the given system what you'll find is

that for most places where you have

dependencies there are tools for

managing those dependencies for you and

very often these systems you might

depend on are stored in what are known

as repositories so repository is just a

collection of things usually related

that you can install that's basically

all a repository is and you might be

familiar with some of them already

right so some examples of repositories

are pi PI which is a well-known

repository for Python packages rubygems

which is similar for ruby crates or io

for rust NPM for nodejs

but other things the repositories too

right like there are positives for

cryptographic keys like key base there

are repositories for system installed

packages like if you ever use the apt

tool in Ubuntu or in Debian you are

interacting with a package repository

where people who have written like

programs and libraries upload them so

that you can then install them

similarly you might have repositories

are entirely open right so the Ubuntu

repositories for example are usually

provided by the Ubuntu developers but in

Arch Linux there might there is

something called the arch user

repository where users can just share

their own libraries and their own

packages themselves. Very often,

repositories are either sort of managed

or they are just entirely open and you

should often be aware of this because if

you're using an entirely open repository

maybe the security guarantees you get

from that are less than what you get in

a controlled repository one thing you'll

notice if you start using repositories

is a very often software is versioned

and what I mean by version well you

might have seen this for stuff like

browsers right where there might be

something like starting like Chrome

version 64 dot zero dot two zero one 903

24 right this is a version number it

might there's a dot here this is one

kind of version number. But, sometimes, if

you start, I don't know, like Photoshop or

you start any other tool, there might be

other kind of versions that are, like,

8.1 dot seven right these version

numbers are usually numerical but not

always sometimes they have hashes in

them for example to refer to git commits

but you might wonder why do we have

these why is it even important that you

add a number to software that you

release the primary reason for this is

because it enables me to know whether my

software would break imagine that I have

a dependency on a library that Jose has

written right and Jose is constantly

doing changes to his library because he

wants to make it better and he decides

that one of the functions that his

library exposes has a bad name so he

renames it my software suddenly stops

working right because I my library calls

a function on Jose's library but that

function no longer exists depending on

which version people have installed of

Jose's library versions helps

because I can say I depend on this

version of Jose's library and there has

to be some rules around what is Jose

allowed to do within a given version if

he makes a change that I can no longer

rely on his version has to change in

some way

there are many thoughts on exactly how

this should work like what are the rules

for publishing new versions how do they

change the version numbers um some of

them are just dictated by time so for

example if you look at browsers they

very often have time versions that look

like this they have a version number on

the far left that's just like which

release and then they have sort of an

incremental number that is usually zero

and then they have a date at the end

right so this is March 24th 2019 for

some reason and usually that will

indicate that this is version 64 of

Firefox from this date and then if they

release sort of patches or hot fixes for

security bugs they might increment the

date but keep the version of the at the

left the same and people have strong

strong opinions on exactly what the

scheme should be and you sort of depend

on knowing what schemes other people use

right if I don't know what scheme Jose

is using for changing his versions

maybe I just have to say you have to run

like eight one seven of Jose's software

otherwise I cannot build my software but

this is a problem, too, right?

Imagine that, Jose, as a responsible

developer of his library, and he finds

the security bug and he fixes it but it

doesn't change the external interfaces

library no functions changed no types

change then I want people to be building

my software with his new version and it

just so happens that building mine works

just fine with his new version because

that particular version didn't change

anything I depended on so one attempted

solution to this is something called

semantic versioning. So, in semantic

versioning, we give each of the numbers

separated by dots in a version number, a

particular meaning. And, we give a

contract for when you have to increment

the different numbers

in particular in semantic versioning we

call this the major version we call this

the minor version and we call this the

patch version and the rules around this

are as follows if you make a change to

whatever your software is and and the

change you made is entirely backwards

compatible right like it does not add

anything it does not remove anything it

does not rename anything externally it

is as if nothing changed then you only

increment the patch number nothing else

so usually security fixes for example

will increment the patch number if you

add something to your library I'm just

gonna call them libraries because

usually libraries are the things where

this matters so for a library if you add

something to the library you increment

the minor version and you set the patch

to zero so in this case if we were to do

a minor release the next minor release

version number would be eight to zero

and the reason we do this is because I

might have a dependency on a feature the

josè added in a 2-0 which means you

can't build my software with eight one

seven that would not be okay

even though if you if I had written it

towards eight one seven you could run it

with a 2-0 the reverse is not true

because it might not have been added yet

and then finally the major version you

increment if you make a backwards

incompatible change where if my software

used to work with whatever version you

had and then you make a change that

means that my software might no longer

work such as removing a function or

renaming it then you increment the major

version and set minor and patch to zero

so the next major version here would be

nine zero zero taken together these

allow us to do really nice things when

setting what our dependencies are in

particular if I depend on a

particular version of someone's library

rather than saying it has to be exactly

this version what I'm really saying is

it has to be the same major version and

at least the same minor version and the

patch can be whatever this means that if

I have a dependency on Jose software

then any later release that is still

within the same major is fine that

includes keep in mind an earlier version

assuming that the minor is the same

imagine that you are saw on some older

computer that has like version eight one

three in theory my software should work

just fine with eight one three as well

it might have whatever bug the Jose

fixed in between like whatever security

issue but this has the nice property

that now you can share dependencies

between many different pieces of

software on your machine if you have

version eight-30 installed and there are

bunch of different software that like

one requires eight one seven one

requires eight two four one requires

eight zero one all of them can use the

same version of that dependencies you

only needed installed once one of the

most common or one of the most familiar

perhaps examples of this kind of

semantic versioning is if you look at

the Python versioning so many of you may

have come across this where python 3 and

python 2 are not compatible with one

another they're not backwards compatible

if you write code in Python 2 you try to

run in Python 3 it might not work there

are some cases where it will but that is

more accidental than anything else and

Python actually follows semantic

versioning at least mostly and so if you

write software that runs on Python 3.5

then it should also work in 3.6 3.7 and

3.8 it will not necessarily work in

Python 4 although that will hopefully be

a long time away but if you write code

for Python 3.5 it will it will possibly

not run on Python 3.4 so one thing you

will see many software projects do is

they try to bring the version

requirements they have as low as

possible if you can depend on major

and then minor in patch zero zero that

is the best possible dependency you can

have because it is completely liberal as

to which version of that major you're

depending on sometimes this is hard

right sometimes you genuinely need a

feature that was added but the lower you

can get the better it is for those who

want to depend on your software in turn

when working with these sort of

dependency management systems or in with

versioning in general you'll often come

across this notion of lock files you

might have seen this where like you try

to do something and it says like cannot

reconcile versions or you get an error

like lock file already exists these are

often somewhat different topics but in

general the notion of a lock file is to

make sure that you don't accidentally

update something the lock file at its

core is really just a list of your

dependencies and which version of them

you are currently using right so my

version string might be eight one seven

and the latest version like on the

internet somewhere might be a three zero

but whatever is installed on my system

is not necessarily one of those two it

might be like eight two four or

something like that and the lock file

will then say dependency josè version

eight to four and the reason you want to

lock file there can be many one of them

is that you might want your builds to be

fast if every single time you try to

build your project whatever tool you

were using download the latest version

and then compile it and then compile

your thing you might wait for a really

long time each time depending on the

release cycle of your dependencies if

you use a lock file then unless the

version

unless you've updated the version in

your lock file it'll just use whatever

it built previously for that dependency

and your sort of development cycle can

be a lot faster another reason to use

lock files is to get reproducible builds

imagine that I produce some kind of

security related software and I very

carefully audited my dependencies and I

produce like a signed binary of like

here is thus like a sworn statement for

me that this

version is secure if I didn't include a

lock file then by the time someone else

installs my program they might get a

later version of their pendency and

maybe that later version as I've been

hacked somehow or just has some other

security vulnerability that I haven't

had a chance to look at yet right and a

lock file basically allows me to freeze

the ecosystem as of this version that I

have checked the extreme version of this

is something called ven during when you

vendor your dependencies it really just

mean you copy/paste of them ven during

means take whatever dependency you care

about and copy it into your project

because that way you are entirely sure

that you will get that version of that

dependency it also means that you can

like make modifications to it on your

own but it has the downsides that now

you no longer get these benefits of

versioning right you no longer have the

advantage that if there are newer

releases of that software your users

might get them automatically like for

example when Hosea fixes his security

issues not that he has any of course one

thing you'll notice is that when talking

about this I've been talking about sort

of bigger processes around your systems

these are things like testing they're

things like checking your dependency

versions

they're also things are just setting up

build systems and often you don't just

want a local build system you want to

build process that includes other types

of systems or you want them to run even

when your computer is not necessarily on

and this is why as you start working a

larger and larger project you will see

people use this idea of continuous

integration and continuous integration

systems are essentially a cloud build

system the idea is that you have your

project stored on the internet somewhere

and you have set it up with some kind of

service that is running an ongoing thing

you for your project whatever it might

be and continuous integration can be all

sorts of stuff it can be stuff like

releasing your library to pi PI

automatically whenever you push to a

particular branch it could be things

like

run your test suite whenever someone

submits a pull request or it could be

check your code style every time you

commit there all sorts of things you

could do with continuous integration and

the easiest way to think about them is

that they're sort of event triggered

actions so whenever a particular event

happens for your a possibly for your

project a particular action takes place

where the action is usually some kind of

script some sequence of programs they're

gonna be invoked and they're gonna do

something this is really an umbrella

term that encapsulate a lot of different

types of services so some continuous

integration services are very general

things like Travis CI or Azure pipelines

or github actions are all very broad CI

platforms they're built to let you write

what you want to happen whenever any

event that you define happens very broad

systems there are some more specialized

systems that deal with things like

continuous integration coverage testing

so like annotate your code and show you

have no tests that test this piece of

code and they're built only for that

purpose or they're built only for

testing browser-based libraries or

something like that and so often you can

find CI tools that are built for the

particular project you're working on or

you can use one of these broader

providers and one thing that's nice is

that many of them are actually free

especially for open source software or

if you're a student you can often get

them for free as well in general the way

you use the CI system is that you add a

file to your repository and this file is

often known as a recipe and what the

recipe specifies is this sort of

dependency cycle again sort of what we

saw with make files it's not quite the

same the events instead of being files

might be something like when someone

pushes a commit or when a commit

contains a particular message or when

someone submits a pull request or

continuously write one example of a

continuous integration service that's

not tied to any particular change to

your code is something called the

dependable

you can find this on github and the

dependent bots is something that you

hook up to your your repository and it

will just scan whether there are newer

versions available of your dependencies

that you're not using so for example if

I was depending on eight one seven and I

had a lock file that locked it to eight

two four and then eight three zero is

released the dependable will go you

should update your log file and then

submit the pull request to your

repository with that update this is a

continuous integration service it's not

tied to me changing anything but to the

ecosystem at large changing often these

CI systems integrate the back into your

project as well so very often these CI

services will provide things like little

badges so let me give an example so for

example here's a project I've worked on

recently that has continuous integration

set up so this project you'll notice

it's readme if I can zoom in here with

that chrome bean nope nope that's much

larger than I wanted here you'll see

that at the top of the the repositories

page they're a bunch of these badges and

they display very various types of

information you'll notice that I have

dependable running right so the

dependencies are currently up to date it

tells me about whether the test suite is

currently passing on the master branch

it tells me how much of the code is

coverage by tests and it tells me what

is the latest version of this library

and what is the latest version of the

documentation of the library that's

available online and all of these are

managed by various continues continuous

integration services another example

that some of you might find useful or

might even be familiar with is the

notion of github pages so github pages

is a really nice service the github

provides which lets you set up a CI

action that builds your repository as a

blog essentially it's it runs a static

site generator called Jekyll

and Jeckle just takes a bunch of

markdown files and then produces a

complete website and that as a part of

get up pages they will also upload that

to get up servers and make it available

at a particular domain and this is

actually how the class website works

class website is not a bunch of like

HTML pages that we manage instead

there's a repository missing semester so

if you look at the missing semester

repository you will see if i zoom out a

little here that this just has a bunch

of markdown files right it has saket

20/20 metaprogramming md so this is the

if I go to raw here this is the raw

markdown for today's lecture so this is

the way that I write the lecture notes

and then I commit that to the repository

we have and I push it and whenever a

push happens the github pages CI is

gonna run the build script for github

pages and produces the website for our

class without me having to do any

additional steps to make that happen and

so yeah sorry good yeah so so Jekyll

it's using a tool called Jekyll which is

a tool that takes a directory structure

that contains markdown files and

produces a website it produces like HTML

files and then as a part of the action

it takes those files and uploads them to

github servers add a particular domain

and usually that's the domain under like

github I oh that they control and then I

have set missing semester to point to

the github domain I want to give you one

aside on testing because it's something

that many of you may be familiar with

from before right you have a rough idea

of what testing is you

run the test before you've seen a test

fail you know like the basics of it or

maybe you've never seen a test fail in

case congratulations but as you as you

get to more advanced projects though

you'll find that people have a lot of

terminology around testing and testing

is a pretty like deep subject that you

could spend many many hours trying to

understand the ins and outs of and I'm

not going to go through it in

excruciating detail but there are a

couple of words that I think it's useful

to know what mean and the first of these

is a test suite so a test suite is a

very straightforward name for all of the

tests in a program it's really just a

suite of tests it's a large collection

of tests that usually are run as a unit

and there are different types of tests

that often make up a test suite the

first of these is what's known as a unit

test a unit test is a often usually

fairly small test of self-contained

tests the tests a single feature what

exactly a feature might mean is a little

bit up to the project but the idea is

that should be sort of a micro test that

only tests a very particular thing then

you have the larger tests that are known

as integration tests integration tests

try to test the interaction between

different subsystems of a program so

this might be something like an example

of a unit test might be if you're

writing an HTML parser to the unit test

might be test that it can parse an HTML

tag an integration test might be here's

an HTML document parse it right so that

is going to be the interview the

integration of multiple of the

subsystems of the parser you also have a

notion of regression tests regression

tests are tests that test things that

were broken in the past so imagine that

someone submits some kind of issue to

you and says your library breaks if I

give it a marquee tag and that makes you

sad so you want to fix it so you fix

your parser to now support marquee tags

but then you want to add a test to your

test suite

the checks that you can parse marquee

tags the reason for this is so that in

the future you don't accidentally

reintroduce that bug. So that is what

a regression tests are for and over time

your project is gonna build up more and

more of these, and they're nice because

they prevent your project from

regressing to earlier bugs. The last one

I want to mention is a concept called

mocking. So mocking is the idea of being

able to replace parts of your system

with a sort of dummy version of itself

that behaves in a way that you control. A

common example of this is you're writing

something that does, oh I don't know, file

copying over SSH. Right? This is a tool

that you've written that does file

copying over SSH there are many things

you might want to mock here. For example,

when running your test suite you

probably don't actually care that

there's a network there. Right? You don't

need to have to like set up TCP ports

and stuff, so instead you're gonna mock

the network. The way this usually works

is that, somewhere in your library, you

have something that like opens a

connection, or reads from the connection,

or writes to the connection, and you're

gonna overwrite those functions

internally in your library with

functions that you've written just for

the purposes of testing, where the read

function just like returns the data, and

the write function just drops the data

on the floor, or something like that.

Similarly, you can write a mocking

function for the SSH functionality. You

could write something that does not

actually do encryption, it doesn't talk

to the network: it just like takes bytes

in here and just magically they pop out

the other side, and you can ignore

everything that's between, because for

the purpose of copying a file, if you

just wanted to test that functionality,

the stuff below doesn't matter for that

test, and you might mock it away. Usually,

in any given language, there are tools

that let you build these kind of mocking

abstractions pretty easily. That is the

end of what I wanted to talk about

metaprogramming, but this is a very, very

broad subject. Things like continuous

integration, build systems, there are so

many out there

that can let you do so many interesting

things with your projects, so I highly

recommend that you start looking into it

a little. The exercises are sort of all

over the place, and I mean that in a good

way. They're intended to try to just show

you the kind of possibilities that exist

for build working with these kind of

processes so for example the last

exercise has you write one of these

continuous integration actions yourself

where it you decide what the event be

and you decide what the action be but

try to actually build one and this can

be something that you might find useful

in your project the example I given the

exercises is try to build an action that

runs like right good or pro Slynt one of

the linters result for the english

language on your repository and if you

do like we could enable that for the

class repository so that our lecture

notes are actually well written right

and this is one other thing that's nice

about this kind of continuous

integration testing is that you can

collaborate between projects if you

write one I can use it in my project and

there's a really handy feature where you

can build this ecosystem of improving

everything any questions about any of

the stuff we record today yeah so the

question is why do we have both make and

see make what do they do and is there a

reason for them to talk together so so

see make I don't actually know what the

tagline for C make is anymore but it's

sort of like a better make for C as the

name implies C make generally

understands the layout of C projects a

little bit better than make Falls do

they're sort of built to try to parse

out what the structure of your

dependencies are what the rules from

going to one to the other is it also

integrates a little bit nicer with

things like system libraries so C may

can do things like detect

given libraries available on your

computer or if it's available at

multiple different paths it tries to

find which of those paths it's present

on on this system and then link it

appropriately so see make is a little

bit smarter than make is make will only

do whatever you put in the make file not

entirely true there are things called

implicit rules that are like built-in

rules in make but they're pretty dumb

whereas emic tries to be able to be a

larger build system that is opinionated

by default to work for C projects

similarly there's a tool called maven so

maven and ant which is another project

they are both built for Java projects

they understand how Java code interacts

with one another how you structure Java

programs and they're built for that task

very often at least when I use make I

use make sort of at the top and then

make my call other tools that build

whatever subsystem they know how to

build right like my make file might call

cargo to build a rust program and then

call see make to build some like see

dependency of that but then at the top

like I'm gonna do some stuff at the end

after the programs have built and that

might just be like run a benchmark which

is in the rust code and then like plot

it using the C code or something like

that right so for me make you sort of

the glue at the top that I might write

usually if your make file gets very

large there's a better tool would you'll

find it like big companies for example

is they often have one build system that

manages all of their software so if you

look at Google for example they have

this open source system called basil and

I don't think Google literally uses

basil inside of Google but it's sort of

based on what they use internally and it

really just its intended to manage the

entire build of everything Google has

and basil in particular is built to be I

think they call it like a polyglot build

framework so the idea is that it works

for many different languages there's

like an implement there's a module for

basil for this language and that

language in that language but they all

integrate with the same basil framework

which then knows how to integrate

dependencies between different libraries

and different languages get a question

sure

so when you say expressions you mean the

things in this file or yeah so these are

so make files are their own language

they are it's a pretty weird language

like it has a lot of weird exceptions in

many ways it's weird just like bash is

weird but in different ways which is

even worse like when you're writing a

make file you sort of you can sort of

think like you're writing bash but

you're not because it's broken in

different ways but but it is its own

language and the way that make files are

generally structured is that you have a

sequence of I think they call them

directives so every like the this thing

oops this thing is a directive and this

is a directive and every directive has a

colon somewhere and everything to the

left of the colon is a target and

everything to the right of the colon is

right of the colon is a dependency and

then all of the lines below that line

are the sequence of operations known as

the rules for once you have the

dependencies how do you build these

targets notice that make is very

particular that you must use a tab to

indent the rules if you do not make will

not work if they must be tabs they

cannot be four eight spaces must be tabs

and like you can have multiple

operations here I like I can do echo

hello or whatever and then they would

first run this and then run this there's

a there's an exercise for today's

lecture that has you try to extend this

make file with a couple of other targets

that you might find interesting that

goes into a little bit more detail

there's also some ability to execute

external commands to like determine what

the dependencies might be if your

dependencies are not like a static list

of files but it's a little limited

usually once you've started needing that

sort of stuff you might want to move to

a more advanced build system yeah so the

question is what happens if I have let's

say that I have library a and library B

and they both depend on library see but

library a depends on like 4.0.1 and

library B depends on 3 dot 4 dot 7 so

they both depend on C and so ideally

we'd like to reuse see but they depend

on different major versions of C what do

we do well what happens in this case

depends entirely on the system that

you're using the language that you're

using in some cases the tool would just

be like well I'll just pick for which

sort of implies that they're not really

using semantic versioning in some cases

the tool will say this is not possible

like if you do this it's an error and

the tool will tell you you either need

to upgrade be like have B use a newer

version of C or you need to downgrade a

you do not get to do this and

compilation will fail some tools are

gonna build two versions of C and then

like when it builds a it will use the

major four version of C and when it

builds B will use the major three

version of C one thing you end up with

is really weird conditions here were

like if C has dependencies then now you

have to build all of C's dependencies

twice to 1 for 3 and 1 for 4 and maybe

they share and maybe they don't you can

end up in particularly weird situations

if imagine that the library see like

imagine that

library see like rights to a file like

rights to some like file on disk some

cache stuff if you run your application

now and like a does something to call

like C dot save and B to something like

C adult load then suddenly your your

application of the bottom is not going

to work because the format is different

right

so these situations are often very

problematic and and most tools that

support semantic versioning will reject

this kind of configuration for exactly

that reason but it's so easy to shoot

yourself in the foot all right we will

see you again tomorrow for security keep

in mind again if you haven't done the

survey the question I care the most

about in the survey is what you would

like us to cover in the last two

lectures so the last two lectures are

for you to choose what you want us to

talk about and to give any questions you

want us to answer so please like add

that if you can and that's it see you

tomorrow