class: left, middle, inverse, title-slide .title[ # GNU Make ] .author[ ### David Benkeser, PhD MPH<br> <span style="font-size: 50%;"> Emory University<br> Department of Biostatistics and Bioinformatics </span> ] .date[ ### INFO550<br><br><svg viewBox="0 0 512 512" style="height:1em;position:relative;display:inline-block;top:.1em;fill:#ffffff;" xmlns="http://www.w3.org/2000/svg"> <path d="M326.612 185.391c59.747 59.809 58.927 155.698.36 214.59-.11.12-.24.25-.36.37l-67.2 67.2c-59.27 59.27-155.699 59.262-214.96 0-59.27-59.26-59.27-155.7 0-214.96l37.106-37.106c9.84-9.84 26.786-3.3 27.294 10.606.648 17.722 3.826 35.527 9.69 52.721 1.986 5.822.567 12.262-3.783 16.612l-13.087 13.087c-28.026 28.026-28.905 73.66-1.155 101.96 28.024 28.579 74.086 28.749 102.325.51l67.2-67.19c28.191-28.191 28.073-73.757 0-101.83-3.701-3.694-7.429-6.564-10.341-8.569a16.037 16.037 0 0 1-6.947-12.606c-.396-10.567 3.348-21.456 11.698-29.806l21.054-21.055c5.521-5.521 14.182-6.199 20.584-1.731a152.482 152.482 0 0 1 20.522 17.197zM467.547 44.449c-59.261-59.262-155.69-59.27-214.96 0l-67.2 67.2c-.12.12-.25.25-.36.37-58.566 58.892-59.387 154.781.36 214.59a152.454 152.454 0 0 0 20.521 17.196c6.402 4.468 15.064 3.789 20.584-1.731l21.054-21.055c8.35-8.35 12.094-19.239 11.698-29.806a16.037 16.037 0 0 0-6.947-12.606c-2.912-2.005-6.64-4.875-10.341-8.569-28.073-28.073-28.191-73.639 0-101.83l67.2-67.19c28.239-28.239 74.3-28.069 102.325.51 27.75 28.3 26.872 73.934-1.155 101.96l-13.087 13.087c-4.35 4.35-5.769 10.79-3.783 16.612 5.864 17.194 9.042 34.999 9.69 52.721.509 13.906 17.454 20.446 27.294 10.606l37.106-37.106c59.271-59.259 59.271-155.699.001-214.959z"></path></svg> <a href="https://bit.ly/info550">.white[bit.ly/info550]</a> <br> <svg viewBox="0 0 448 512" style="height:1em;position:relative;display:inline-block;top:.1em;fill:#ffffff;" xmlns="http://www.w3.org/2000/svg"> <path d="M448 360V24c0-13.3-10.7-24-24-24H96C43 0 0 43 0 96v320c0 53 43 96 96 96h328c13.3 0 24-10.7 24-24v-16c0-7.5-3.5-14.3-8.9-18.7-4.2-15.4-4.2-59.3 0-74.7 5.4-4.3 8.9-11.1 8.9-18.6zM128 134c0-3.3 2.7-6 6-6h212c3.3 0 6 2.7 6 6v20c0 3.3-2.7 6-6 6H134c-3.3 0-6-2.7-6-6v-20zm0 64c0-3.3 2.7-6 6-6h212c3.3 0 6 2.7 6 6v20c0 3.3-2.7 6-6 6H134c-3.3 0-6-2.7-6-6v-20zm253.4 250H96c-17.7 0-32-14.3-32-32 0-17.6 14.4-32 32-32h285.4c-1.9 17.1-1.9 46.9 0 64z"></path></svg> <a href="https://benkeser.github.io/info550/readings#gnu-makefiles">.white[Additional reading]</a> ] --- <!-- Zip up everything in /example so available for download link --> <style type="text/css"> .remark-slide-content { font-size: 25px } </style> ### Why GNU Make? GNU Make is a program often used for compiling software. * Building up many dependent pieces of software. But that's exactly __what data analysis is__! * Analysis report &#8592; tables and figures &#8592; code &#8592; data GNU Make is a formal system for: * automating a workflow * just type `make` and hit enter! * documenting a workflow * documenting dependencies among data, code, results. ??? If you can effectively use `Make`, you are living the reproducible dream. Literally you will just go to the command line and type `make` and your whole analysis will be off and running. Data get updated? Swap in the new data set and `make`. Change the code for Figure 1? `make`. It takes time to get to this level, but this is the goal of reproducible data science. Remember, everything that we're doing is about making our lives easier *in the long run*. It might be convenient in the short term to: * keep all your code in one long file, * open the file and interactively execute code, * copy/paste results. But in the long run, you'll have to * sift through 1000s of lines of code every time you want to make a small change, * remember the exact sequence of code that you executed six months or a year ago, * copy/paste results 100 times! --- ### How does it work? In your project directory, you'll have a plain text `Makefile`. ````markdown # here's a recipe for target target: prereq [TAB] action1 [TAB] action2 ```` * use `#` for comments. * `target` and `prereq` are filenames. * `action1` and `action2` are commands. * `[TAB]` denotes that you need tab rather than multiple spaces. At the command line, in the project directory, we type ```bash make ``` ??? If you ever see the error `Makefile:3: *** missing separator. Stop.`, you forgot to use tabs. This is going to be an annoying aspect of this lecture -- copying and pasting code from the slides to a text editor is not guaranteed to respect tabs. --- ### How does it work? GNU Make logic: * Want to update `target`, which depends on `prereq`. * Does `target` exist? * If <em style="color:#da291c";>no</em>, execute `action1`, `action2` * If <em style="color:#348338";>yes</em>, has `prereq` changed since `target` was last updated? * If <em style="color:#348338";>yes</em>, execute `action`s. * If <em style="color:#da291c";>no</em>, take no action, `target` is already up to date. __What if `prereq` itself depends on other files?__ --- ### Graph of dependencies ````markdown # here's a recipe for target target: prereq [TAB] action1 [TAB] action2 # here's a recipe for prereq prereq: prereq2 [TAB] action3 ```` If we type `make`, we will make `target`, the first entry in `Makefile`. * Or, equivalently, `make target`. * But, we can also use `make prereq`. ??? If we `make prereq`, the logic is the same as before. We check for existence of `prereq` if it doesn't exist, we take `action3`. If it does, we check whether `prereq2` has been updated. If so, we take `action3`. If <em style="color:#da291c";>no</em>, then `prereq` is already up to date. --- ### Graph of dependencies `make prereq` logic: * Want to update `prereq`, which depends on `prereq2`. * Does `prereq` exist? * If <em style="color:#da291c";>no</em>, execute `action3` * If <em style="color:#348338";>yes</em>, has `prereq2` changed since `prereq` was last updated? * If <em style="color:#348338";>yes</em>, execute `action3`. * If <em style="color:#da291c";>no</em>, take no action, `prereq` is already up to date. Same as before, but now for `prereq` rather than `target`. --- ### Graph of dependencies `make`/`make target` logic: .small[ * Want to update `target`, which depends on `prereq`. * Need to update `prereq`, which depends on `prereq2`. * Does `prereq` exist? * If <em style="color:#da291c";>no</em>, execute `action3` * If <em style="color:#348338";>yes</em>, has `prereq2` changed since `prereq` was last updated? * If <em style="color:#348338";>yes</em>, execute `action3`. * If <em style="color:#da291c";>no</em>, take no action, `prereq` is already up to date. * Does `target` exist? * If <em style="color:#da291c";>no</em>, execute `action1`, `action2`. * If <em style="color:#348338";>yes</em>, has `prereq` changed since `target` was last updated? * If <em style="color:#348338";>yes</em>, execute `action1`, `action2`. * If <em style="color:#da291c";>no</em>, take no action `target` is already up to date. ] --- ### Example Consider the files in <a href="example.zip" download>`example.zip`</a>. ```bash ls ``` ``` ## Makefile ## cleandata.R ## raw_data.txt ## report.Rmd ``` ```bash cat Makefile ``` ``` ## # rule for making report ## report.html: data.txt report.Rmd ## Rscript -e "rmarkdown::render('report.Rmd', quiet = TRUE)" ## ## # rule for cleaning data ## data.txt: cleandata.R raw_data.txt ## chmod +x cleandata.R && \ ## Rscript cleandata.R ``` ??? In general, I will encourage you to have separate folders for R code and raw/processed data, but for now let's allow bad habits. We see that `Makefile` ultimately wants to compile `report.html` using R Markdown. The html report is generated from the `report.Rmd` script and additionally depends on the data set `data.txt`, which itself has depends on `cleandata.R` and `raw_data.txt`. `data.txt` is made by executing the `cleandata.R` script. In the next slide we'll see the contents. --- ### Example Let's pause for a moment on the structure of the rule ````markdown data.txt: cleandata.R raw_data.txt chmod +x cleandata.R && \ Rscript cleandata.R ```` * `make` executes each action in a separate shell * `cd dir` on one line, execute file in `dir` on next; .red[will not work!] * from docs: .red[cannot] expect actions to be performed in sequence * use `&& \` breaks commands over multiple lines * commands separated by `&&` run sequentially with each running only if the last does not fail * `\` is an escape character; bash ignores the new line --- ### Example `cleandata.R` `````r #! /usr/bin/env Rscript # read data raw_data <- read.csv("raw_data.txt", header = TRUE) # remove NAs data <- raw_data[!is.na(raw_data), , drop = FALSE] # save data write.table(data, "data.txt", quote = FALSE, row.names = FALSE) ```` * Removes missing data and saves as `data.txt`. * `make data.txt` results in a cleaned data set being saved. ??? The idea here is to illustrate what a typical `make` workflow could look like. Obviously data cleaning in real life is more complex than this. --- ### Example `report.Rmd` ````markdown --- title: "Analysis Report" output: html_document --- `r ''````{r, read-data, echo = FALSE} data <- read.table("data.txt", header = TRUE) ````` There were `r nrow(data)` data points with mean `r mean(data$X)`. ```` * reads in `data.txt`, reports simple summary statistics ??? Of course, we could have included everything in the `.Rmd` document. In this simple example, there's really not a price to pay for that. However, as the complexity of the analysis increases, there becomes more of a price to pay: * the `report.Rmd` document itself is difficult to read * if something breaks, its harder to debug where * finding specific lines of code becomes difficult In the next chapter, we'll discuss project management and why it's a good idea to split up workflows into multiple chunks. --- ### Breakout exercise * Add a script `make_fig1.R` to the `example` directory (from <a href="example.zip" download>`example.zip`</a>) * Read in `data.txt` * Save `.png` of a histogram of the data (see below) * Add a `make` rule for `fig1.png`. * Update the `report.Rmd` to include `fig1.png` (see below). * Update `make` rule for `report.html` * Compile `report.html` using `make` ```r # assuming you want a histogram of x png("fig1.png") hist(data$X) dev.off() ``` ````markdown  ```` --- ### `PHONY` rules Sometimes we wish to add rules that do not create files. ````markdown # rule for making report report.html: data.txt report.Rmd Rscript -e "rmarkdown::render('report.Rmd', quiet = TRUE)" # rule for cleaning data data.txt: cleandata.R raw_data.txt chmod +x cleandata.R && \ Rscript cleandata.R # clean up directory clean: rm report.html data.txt ```` ??? Can be nice to clean up the working directory to recreate all output of the workflow explicitly. Notice `clean` has no dependencies. --- ### `PHONY` rules ```bash make report.html ``` ``` ## chmod +x cleandata.R && \ ## Rscript cleandata.R ## Rscript -e "rmarkdown::render('report.Rmd', quiet = TRUE)" ``` ```bash make clean ls ``` ``` ## rm report.html data.txt ## Makefile ## cleandata.R ## raw_data.txt ## report.Rmd ``` --- ### `PHONY` rules Works fine .red[as long as there's no file named] `clean`. ```bash touch clean make clean ``` ``` ## make: `clean' is up to date. ``` In this case, we need to tell `make` that clean is a `PHONY`. ````markdown # clean up directory .PHONY: clean clean: rm report.html data.txt ```` ??? Something to be aware of. You can generally leave off `PHONY` but good to know about it. For example, I could see a `make docs` command and a `docs` folder containing documentation. --- ### Automatic variable names Code can be shortened using special characters. * In a rule, * `$@` can be used in place of the target; * `$^` can be used in place of all dependencies; * `$<` can be used in place of the first dependency. ````markdown # rule for making report report.html: data.txt report.Rmd Rscript -e "rmarkdown::render('report.Rmd', output_name = '$@', quiet = TRUE)" # rule for cleaning data data.txt: cleandata.R raw_data.txt chmod +x $< && \ Rscript $< ```` --- ### Pattern Rules What if we want rules for multiple figures? * A general rule of coding: .red[don't repeat yourself]. ````markdown # rule for figure 1 fig1.png: make_fig1.R data.txt Rscript make_fig1.R # rule for figure 2 fig2.png: make_fig2.R data.txt Rscript make_fig2.R ... ```` ??? If you ever find yourself copy/pasting code and changing one number per line, ask yourself if there's a smarter way to do things. --- ### Pattern Rules To simplify, we can use __pattern rules__. * `%` is a wildcard * `$*` references the matched part of the target * e.g., if we say `make fig1.png` then `$*` evaluates to `fig1`. ````markdown # rule for making figures %.png: make_%.R data.txt Rscript $*.R # PHONY rule for all figures .PHONY: figs figs: fig1.png fig2.png ```` If we execute `make figs`, we will make `fig1.png` and `fig2.png` according to the rule for making figures. .small[line 3 above could equivalently be written `Rscript $<`] --- ### Documenting rules It can be useful to create `help` targets in your Makefiles. ````markdown .PHONY: help help: @echo "report.html : Generate final analysis report." @echo "data.txt : Clean raw_data.txt by removing NAs." @echo "fig1.png : Make a histogram of X." @echo "clean : Remove cleaned data and report." ```` Then we can execute ```bash make help ``` ``` ## report.html : Generate final analysis report. ## data.txt : Clean raw data by removing NAs. ## fig1.png : Make a histogram of X. ## clean : Remove cleaned data and report. ``` ??? In `make`, adding `@` in front of a command means that the command statement is not printed out. --- ### Self-documenting rules Every time we add/remove rule, we need to add an action to `help`. We can be .green[more clever] about this. ````markdown .PHONY: help help: Makefile @sed -n 's/^##//p' $< ```` A lot to unpack here: * a rule that depends on `Makefile` itself * use bash `sed` to look in `Makefile` (i.e., `$<`) for the string `##` and print every line on which that string is found. * `-n` means that the `##` themselves will not be printed --- ### Self-documenting rules ````markdown ## report.html : Generate final analysis report. # using the cleaned data, generate the report report.html: data.txt report.Rmd Rscript -e "rmarkdown::render('report.Rmd', quiet = TRUE)" ## data.txt : Clean raw_data.txt by removing NAs. data.txt: cleandata.R raw_data.txt chmod +x cleandata.R && \ Rscript cleandata.R ## fig1.png : Make a histogram of X. fig1.png: make_fig1.R data.txt Rscript make_fig1.R ## clean : Remove compiled report and cleaned data. clean: rm report.html data.txt .PHONY: help help: Makefile @sed -n 's/^##//p' $< ```` ??? Note that you can still make comments that do not appear in `help` by using a single `#`. It seems possible that with a slightly more complex call to `sed`, you could even automate grabbing the name of the target for the help file. Though this is not entirely straightforward and wouldn't really represent that much time savings. Nevertheless, if someone figures out how to do it, let me know! --- ### Final notes on make * If you name your make file `something_else`, you can use `make -f something_else`. * Can be helpful to have an `all: target1 target2 target3` at the top of your `Makefile`. * That way typing `make` builds everything. * You can get quite fancy with `make`. * Define [variables](https://swcarpentry.github.io/make-novice/06-variables/index.html) * Define [functions](https://swcarpentry.github.io/make-novice/07-functions/index.html) --- ### Breakout exercise * Download <a href="exercise.zip" download>`exercise.zip`</a>, which contains * `clinical.txt`, a made up clinical data set * `biomarkers.txt`, a made up data set consisting of two biomarkers * `biom_report.Rmd`, a made up R Markdown report * Analyze `biom_report.Rmd` to understand what it is doing * Create a more legible workflow based on `make`. * Goals: * make `biom_report.Rmd` more human readable; * separate data cleaning and figure construction; * document dependencies in `Makefile`.