Read the learning objectives listed on this page: Bear in mind what you’re trying to achieve while reading this text. If a topic isn’t making much sense, and isn’t in the objectives, you can probably skip over it. If a topic is listed in the objectives, you should keep studying it until you are confident you’ve met the objective.

Check you have achieved each learning objective listed on this page. If you’re not sure about any of them, ask in Slack.

Exercise

Label each operator with its correct arity:

In class, we will have a discussion about each of the below prompts. Think about them, and write down notes in your notebook to help you participate in the discussion.

Variables are names for memory locations 🧶 🧶 Memory location Memory is where we store values. You can think of memory as a long list of locations. Each location can store one byte of data. We can store the byte 0x41 in a memory location. One variable always points at one memory location, but depending on its type may include the data from subsequent memory locations too. . What’s inside the memory location can be changed.

Some programming languages are compiled. Others are interpreted.

Read the learning objectives listed on this page: Bear in mind what you’re trying to achieve while reading this text. If a topic isn’t making much sense, and isn’t in the objectives, you can probably skip over it. If a topic is listed in the objectives, you should keep studying it until you are confident you’ve met the objective.

Key take-aways:

• Most programs take some input, and produce some output.
• Instead of writing one program to do exactly what we want, we can often combine existing programs.
• Programs read from stdin, and write to stdout and stderr.
  • They write their main output to stdout.
  • They write error messages, progress messages, and other information that isn’t their main output to stderr.
• We can pass information between programs using a pipe: |
• We can write the output of programs to a file using >, or append to a file using >>.

Comparison with JavaScript

A shell pipeline is like a chain of function calls.

In JavaScript we joined together pipeline stages (functions) with .s:

["Hello", "I", "am", "EXTRA", "happy"]
    .filter((word) => /[A-Z]/.test(word))
    .map((word) => word.toLowerCase())
    .sort()

In shell we join together pipeline stages (commands) with |s:

printf "Hello\nI\nam\nEXTRA\nhappy\n" | grep '[A-Z]' | tr '[:upper:]' '[:lower:]' | sort

Next we will learn about some programs commonly used in pipelines.

We’ve already used grep to search for text in files using regular expressions.

We can also pipe other commands’ output to grep to search the output the same way.

For example, we can write:

% ls -1
report-draft
report-version-1
report-version-1.1
report-version-2
report-final
report-final-2
% ls -1 | grep -v '[0-9]'
report-draft
report-final

The original ls -1 command showed us all the files in the current directory.

By piping this to grep -v '[0-9]' we can filter this output down to just the files whose names don’t contain numbers.

grep operates on lines, and ls -1 outputs one file per line, so grep tests each file one at a time.

ls vs ls -1

In our terminal, when we run ls -1, we get one file output per line. But if we run ls in our terminal, we get the files on one line, separated by spaces.

We know that grep operates on individual lines, so it may seem like ls | grep would have a problem - ls prints more than one file per line.

But ls behaves specially. It detects whether it’s outputting to a terminal, or a pipe, and acts differently:

• If it’s outputting to a pipe, it outputs one file per line.
• If it’s outputting to a terminal, it tries to be useful and take up less space. But if you pass -1 it will force ls to output one file per line.

So we can write ls | grep -v '[0-9]' - we don’t need to pass -1 to ls.

It’s good to know that sometimes programs behave differently when outputting to a terminal or a pipeline.

sort sorts its input. uniq deduplicates adjacent matching lines.

(Source, including text-only transcript: https://wizardzines.com/comics/sort-uniq/)

Learn about sort and uniq from their man pages (and the backlog exercises).

Often we pipe to sort | uniq not just uniq so that duplicate lines will be next to each other before they’re passed to uniq.

For the following quizzes, consider the following input file:

% cat input.txt
pigs 10
chickens 2
pigs 10
goats 3
hamsters 300

head outputs lines (or bytes) from the start of a file. tail outputs lines (or bytes) from the end of a fail.

(Source, including text-only transcript: https://wizardzines.com/comics/head-tail/)

Learn about head and tail from their man pages (and the backlog exercises).

Imagine we have an input file which has 100 lines.

tr translates (replaces) characters.

Learn about tr from its man page (and the backlog exercises).

All of the tools we’ve seen so far operate on lines, words, or characters.

Often these are the formats we have. And indeed, often we decide to have our programs output in these formats to make the output easy to process with tools.

But there are more complex formats it can be useful to process too. You have already used JSON in the course. It is a text format that allows us to represent arrays, objects, strings, numbers, and more.

jq is a tool for processing JSON without having to write a whole program. This can be really useful to quickly analyse some data.

Practice using jq using the relevant backlog exercises.