Shell Scripting and Automation Basics

Every system administrator eventually faces tasks that must be done repeatedly: rotating log files, copying backups to a remote server, provisioning user accounts. When you perform these by hand, three problems emerge. First, the work is tedious; Google’s SRE team calls this kind of repetitive, automatable operational work toil. Second, manual work is inconsistent; you will eventually forget a step or mistype a path. Third, manual processes are not repeatable in any verifiable way.

Shell scripts solve all three problems. A script encodes the exact sequence of commands, runs them the same way every time, and serves as living documentation. This lecture covers the anatomy of a well-structured Bash script: how variables and quoting prevent subtle failures, how the Unix I/O model’s exit code contract governs how scripts detect and propagate failure, how conditionals and loops let a script react to system state, how functions organize reusable logic, and how a handful of safe-scripting defaults catch the majority of bugs at the point of failure. The lecture also covers scheduling with cron, regular expressions and the tools that use them, and the point at which Bash’s strengths run out and a more capable language should take over. The mechanics of pipes, redirection, and stream handling are covered in the Terminal and Shell practicality; this lecture focuses on what those primitives mean for script design.

Script Structure and the Shebang

A shell script is simply a text file containing commands that the shell can execute, processed top to bottom by the interpreter. What separates a robust script from a fragile one is rarely the commands themselves; it is the structure around them: declaring the right interpreter so the script runs correctly on any machine, controlling execution permissions so the kernel knows the file is meant to be run, and understanding the difference between invoking a script directly and sourcing its contents into the current shell. The following script copies a web directory to a backup location, and the first line is already doing significant work:

#!/usr/bin/env bash
cp -r /var/www/html /backups/html-backup

The first line is the shebang (sometimes called a hashbang). It tells the operating system which interpreter should run the file. Using #!/usr/bin/env bash instead of #!/bin/bash is more portable, because env searches your PATH for bash rather than assuming a fixed location.

Save the file as backup.sh and make it executable with chmod +x backup.sh. Without that permission, the kernel will refuse to execute the file directly. You can also invoke any script explicitly with bash backup.sh, which bypasses the permission check, but making scripts executable is the conventional approach: it communicates intent and allows the shebang line to select the right interpreter.

The shebang mechanism is not exclusive to Bash. A Python script beginning with #!/usr/bin/env python3 uses the same kernel mechanism, which means Python scripts can be made executable and scheduled with cron using exactly the same approach. The choice of interpreter lives in the shebang line; the rest of the Unix tooling treats the script identically regardless of what is behind it.

Historical Note bash Is a Pun, and the Shebang Has a Name (1979-1989)

The first widely-used Unix shell was written by Stephen Bourne at Bell Labs in 1979. When the GNU Project needed a free replacement in 1989, Brian Fox wrote the Bourne-Again Shell (bash) as a deliberate pun on Bourne’s name. bash added command history, tab completion, job control, and arrays to the Bourne shell’s feature set and became the default shell on most Linux distributions. The shebang line (#!) predates bash: it was added to the Unix kernel in 1980 so that scripts could declare their own interpreter rather than relying on the calling shell to guess. The name “shebang” is a contraction of “hash bang.” macOS used bash as its default shell until Catalina (2019), when Apple switched to zsh, not for technical reasons but because bash 3.2 is the last version under the permissive GPLv2 license, and Apple preferred not to ship the newer GPLv3-licensed versions.

Variables and Quoting

Hardcoding paths like /var/www/html makes a script fragile. Variables let you change a value in one place and have it take effect everywhere. Assigning a variable uses the bare name; reading it back requires a $ prefix, which tells the shell to expand the variable to its current value before passing the result to a command:

#!/usr/bin/env bash
SOURCE="/var/www/html"
DEST="/backups/html-backup"
cp -r "$SOURCE" "$DEST"

Notice the double quotes around $SOURCE and $DEST. Quoting is one of the most important habits in shell scripting. If a path ever contains a space (for example, /var/www/my site), an unquoted $SOURCE would split into two separate arguments and break the command. The rule of thumb is simple: always double-quote variable expansions.

Double quotes allow variable expansion and command substitution; single quotes treat everything literally. "Hello $NAME" expands $NAME; 'Hello $NAME' prints the literal string $NAME.

Command substitution lets you capture the output of a command into a variable. This is invaluable for timestamps, hostnames, and dynamic paths:

TIMESTAMP=$(date +%Y%m%d-%H%M%S)
DEST="/backups/html-backup-$TIMESTAMP"
cp -r "$SOURCE" "$DEST"

Now every run creates a uniquely named backup directory. The $(...) syntax is the modern form. The older backtick form (TIMESTAMP=`date +%Y%m%d-%H%M%S`) does the same thing but cannot be nested without escaping, which makes it harder to read in complex expressions.

A related construct, the here document (heredoc), feeds a block of inline text to a command as standard input. It is not a quote type in the same sense as single or double quotes; it is an input-redirection form whose body may either expand shell syntax or preserve it literally depending on how you write the delimiter. The << operator introduces the block, followed immediately by a delimiter word of your choosing; the block ends when that word appears alone on a line. EOF is only a conventional delimiter name, short for “end of file”; you could use END, REPORT, or any other word that does not appear by itself inside the body. The important part is that Bash looks at the delimiter word on the opening line to decide how to treat the entire heredoc body. That is why the quotes, if any, appear around EOF rather than around the block itself. If the delimiter is quoted, Bash does not perform parameter expansion, command substitution, or arithmetic expansion inside the body. If the delimiter is unquoted, Bash performs those expansions before passing the text to the command:

SOURCE="/var/www/html"

cat > /tmp/literal-report.txt << 'EOF'
Source: $SOURCE
Generated: $(date)
Check: $(( 2 + 2 ))
EOF

cat > /tmp/expanded-report.txt << EOF
Source: $SOURCE
Generated: $(date)
Check: $(( 2 + 2 ))
EOF

The first block writes the characters $SOURCE, $(date), and $(( 2 + 2 )) literally into the file. The second writes the expanded values instead. Using nearly identical bodies is the important teaching point here: the only difference is whether the delimiter is quoted. Heredocs are the idiomatic way to generate multi-line input from within a script without creating a temporary file or chaining multiple echo calls.

For example, if date returned Tue Apr 21 15:30:00 UTC 2026, inspecting the two files would show:

$ cat /tmp/literal-report.txt
Source: $SOURCE
Generated: $(date)
Check: $(( 2 + 2 ))

$ cat /tmp/expanded-report.txt
Source: /var/www/html
Generated: Tue Apr 21 15:30:00 UTC 2026
Check: 4

If the order of > and << looks surprising, Bash also accepts cat << EOF > /tmp/expanded-report.txt and treats it the same way here. << EOF connects the command’s standard input, while > /tmp/expanded-report.txt redirects the command’s standard output. Because those redirections affect different streams, Bash can set them both up before it runs cat, and the order does not change the result in this example. By contrast, the order does matter when two redirections interact with the same stream, such as 2>&1.

Script Arguments

A script that hardcodes its inputs is a one-off command, not a reusable tool. Accepting arguments at the call site is what makes a script general-purpose: the same script can be called with different paths, different targets, or different modes without ever editing the file. There is an important distinction in terminology here: an argument is the value provided when a script is called (./backup.sh /var/www /backups); a parameter is how the script accesses that value internally. Bash exposes arguments through positional parameters.

The positional parameters are $1, $2, $3, and so on, in the order the arguments were passed. $0 holds the name of the script itself. $# holds the total count of arguments passed. $@ expands to all arguments as separate quoted words, which is the correct form to use when passing them to another command:

#!/usr/bin/env bash
if [[ "$#" -lt 2 ]]; then
    echo "Usage: $(basename "$0") <source> <destination>" >&2
    exit 1
fi

SOURCE="$1"
DEST="$2"
cp -r "$SOURCE" "$DEST"

Checking $# before doing any work is a good habit. If the count is wrong, the script exits immediately with a usage message rather than failing halfway through with a confusing error about an empty variable. Printing the usage message to stderr (with >&2) keeps it separate from normal output so it does not corrupt pipelines or log files. The $(basename "$0") call strips any leading path from the script name, so the usage line reads backup.sh rather than /usr/local/bin/backup.sh or ./backup.sh depending on how it was invoked.

Environment Variables

Shell variables live only within the running script. When a script calls another program, that program gets a copy of the environment: a set of key-value pairs passed from parent to child every time a new process starts. This is a property of the Unix process model: every process inherits its parent’s environment automatically. Variables are added to the environment only when you mark them with the export keyword:

SCRIPT_NAME="backup"

export BACKUP_ROOT="/backups"
export AWS_REGION="us-west-2"

SCRIPT_NAME is a shell variable: other programs your script launches will not see it. BACKUP_ROOT and AWS_REGION are environment variables: any command invoked by the script inherits them. The distinction matters whenever your script calls a tool that reads configuration from the environment, such as the aws CLI, git, or rsync. Your script also inherits the environment of whatever launched it, which is why variables set in your shell profile are available in scripts you run interactively.

PATH in non-interactive contexts. The PATH variable lists directories the shell searches when you type a command without a full path. Your interactive shell has a rich PATH built from profile files. Cron often sets a minimal PATH, typically just /usr/bin:/bin. systemd services also run with a controlled environment that often differs from your interactive shell, though the exact PATH depends on the unit and the distribution. This is why a script that works perfectly in your terminal can fail under cron or systemd with “command not found”: the command exists on the system, but the service environment does not include its directory. The reliable fix is to use absolute paths everywhere in scripts intended for non-interactive use, or to set PATH explicitly at the top of the script. The env command prints the full current environment, which makes it a useful diagnostic when a script behaves differently across contexts: run env in both an interactive shell and a cron job to see what is missing.

Sourcing versus executing. Running ./script.sh spawns a child process that inherits a copy of your environment, executes the script, and exits. Any variables the script sets or exports are discarded with that process when it returns. Running source script.sh (or equivalently . script.sh) runs the script’s commands directly in the current shell, so any export statements take effect in your working session. Environment setup scripts, the kind that configure your shell for a project or a deployment, are always sourced rather than executed for exactly this reason.

Scripts and the Unix I/O Model

The mechanics of pipes, redirection, and streams are covered in the Terminal and Shell practicality. What matters for scripting is understanding the design philosophy behind the model and what it implies for how a well-structured script behaves.

Unix programs communicate through three standard streams: stdin for input, stdout for normal output, and stderr for error messages. The reason these are separate is composability. A program that sends errors to stdout corrupts any pipeline it sits in, because the downstream program now receives a mix of data and error messages it was not designed to handle. A program that sends errors to stderr keeps the pipe clean: stdout carries the data, stderr carries the diagnostics, and a caller can route each to the right place independently. A script should follow the same convention, sending all error messages to stderr:

echo "Error: source directory does not exist" >&2
exit 1

Exit codes are the contract. Every command finishes with an exit code: 0 for success, any non-zero value for failure. This is how pipeline stages and calling scripts detect whether a command worked. You can test a command’s success directly in an if statement:

if ! cp -r /var/www/html /backups/html-latest; then
    echo "Backup failed" >&2
    exit 1
fi

A script that exits with 0 when it fails is lying to whoever called it. Cron, systemd, CI systems, and other scripts all use exit codes to detect failure and decide whether to alert, retry, or abort. Getting exit codes right is not optional for production scripts.

Conditionals

Scripts that run on a schedule or in the background cannot rely on a human to notice when something goes wrong. The source directory might have been moved, the target disk might be full, or a required tool might be missing. Conditionals let a script interrogate the state of the system before acting, either proceeding with confidence or halting with a specific, actionable error rather than producing partial, silent, or misleading output. This is the difference between a script that fails mysteriously at step seven and one that tells you at step one why it cannot proceed. Bash’s [[ ]] test construct and if statement are the primary tools for this.

#!/usr/bin/env bash
SOURCE="/var/www/html"
TIMESTAMP=$(date +%Y%m%d-%H%M%S)
DEST="/backups/html-backup-$TIMESTAMP"

if [[ -d "$SOURCE" ]]; then
    cp -r "$SOURCE" "$DEST"
    echo "Backup complete: $DEST"
else
    echo "Error: source directory $SOURCE does not exist" >&2
    exit 1
fi

The [[ ... ]] construct is Bash’s extended test command. It is preferred over the older [ ... ] (which is actually the test command) because it handles quoting more gracefully and supports pattern matching. The following file tests are the ones you will reach for most often:

Test	Meaning
`-d path`	True if path is a directory
`-f path`	True if path is a regular file
`-e path`	True if path exists at all
`-r path`	True if path is readable
`-w path`	True if path is writable
`-s path`	True if file is non-empty

String comparisons use == and != inside [[ ]], while numeric comparisons use -eq, -ne, -lt, -gt, -le, and -ge. You can chain conditions with elif:

if [[ "$EUID" -ne 0 ]]; then
    echo "This script must be run as root" >&2
    exit 1
elif [[ ! -d "$SOURCE" ]]; then
    echo "Source directory missing" >&2
    exit 1
else
    echo "Preconditions met, proceeding..."
fi

Loops

A script that operates on a single, hardcoded target is useful; one that operates on a list of targets is a tool. Loops are what make the difference: the same logic applies to every item in a collection without duplicating the code, the collection can grow or shrink without touching the script’s structure, and new items are handled automatically. This composability is central to how professional automation is written. Bash provides several loop constructs suited to different iteration patterns. The for loop is the most common, iterating over a list of values:

#!/usr/bin/env bash
TIMESTAMP=$(date +%Y%m%d-%H%M%S)
DIRS=("/var/www/html" "/etc/nginx" "/etc/ssh")

for dir in "${DIRS[@]}"; do
    if [[ -d "$dir" ]]; then
        dest="/backups/$(basename "$dir")-$TIMESTAMP"
        cp -r "$dir" "$dest"
        echo "Backed up $dir to $dest"
    else
        echo "Skipping $dir (not found)" >&2
    fi
done

The "${DIRS[@]}" syntax expands the array so that each element is treated as a separate word, even if an element contains spaces. The basename command strips the leading path components, turning /var/www/html into just html.

While loops are useful when you need to read input line by line, such as processing a configuration file:

while IFS= read -r dir; do
    [[ -z "$dir" || "$dir" == \#* ]] && continue
    echo "Processing: $dir"
done < /etc/backup-dirs.conf

IFS= prevents leading and trailing whitespace from being trimmed, and -r prevents backslash interpretation. The continue statement skips blank lines and lines starting with #.

Arithmetic loops. Bash’s (( ... )) construct evaluates integer arithmetic and supports C-style operators. It returns exit code 0 when the arithmetic result is non-zero, and exit code 1 when the result is zero, making it a natural fit for loop conditions and counters:

#!/usr/bin/env bash
i=0
while (( i < 5 )); do
    printf "i = %d\n" "$i"
    (( i++ ))
done

The same construct handles all integer arithmetic: (( count += 1 )), (( total = a + b )), and so on. This is the preferred approach over the older expr command, which requires a command substitution and is significantly slower. The exit-code behavior of (( )) has one practical consequence covered in the Safe Scripting Defaults section.

Functions

As a script grows, the same pattern tends to recur: validate that something exists, perform an operation on it, log the outcome. Without functions, that pattern gets copied and pasted wherever it is needed, and a bug fix or improvement has to be applied everywhere the copy appears. Functions solve this by naming a block of logic so it can be called like any other command. They accept arguments, communicate success or failure through an exit code, and can use local variables so internal names do not accidentally overwrite global state. A Bash function does not run in a separate process, so it shares the shell’s working directory and most shell state with the caller. A well-factored Bash script reads much like a program in any other language: a main block that drives the logic, and named functions that handle the reusable parts.

#!/usr/bin/env bash
TIMESTAMP=$(date +%Y%m%d-%H%M%S)

backup_dir() {
    local source="$1"
    local dest="/backups/$(basename "$source")-$TIMESTAMP"

    if [[ ! -d "$source" ]]; then
        echo "WARN: $source not found, skipping" >&2
        return 1
    fi

    cp -r "$source" "$dest"
    echo "OK: $source -> $dest"
    return 0
}

backup_dir "/var/www/html"
backup_dir "/etc/nginx"
backup_dir "/etc/ssh"

Several things are worth noting here. The local keyword restricts a variable’s scope to the function; without it, variables are global by default, which leads to subtle bugs in larger scripts. Function arguments are accessed as $1, $2, and so on, using the same syntax as script arguments. The return statement sets the function’s exit status: 0 for success, non-zero for failure. This is distinct from exit, which terminates the entire script.

Safe Scripting Defaults

By default, Bash does not stop when a command fails. If a cp returns a non-zero exit code, the script silently continues to the next line. If you mistype a variable name, Bash substitutes an empty string rather than flagging the mistake. If the first command in a pipeline fails, the pipeline still reports success as long as the last command succeeds. Each of these defaults has a historical justification rooted in interactive shell use, but each one is also a common source of bugs in scripts, producing failures that cascade far from their origin and are difficult to diagnose. Bash provides a set of options, conventionally set at the top of every production script, that override these defaults:

#!/usr/bin/env bash
set -euo pipefail

Each option addresses a distinct failure mode. set -e (errexit) causes the script to exit on an unhandled non-zero status. Bash has important exceptions here: commands used in if, while, until, &&, ||, and ! contexts are being tested rather than treated as fatal. Without set -e, a failing cp is silently ignored and the script keeps running. set -u (nounset) causes the script to exit if you reference an undefined variable, catching typos like $SORUCE instead of $SOURCE. set -o pipefail changes pipeline semantics: without it, cmd1 | cmd2 succeeds as long as cmd2 succeeds, even if cmd1 fails; with pipefail, the pipeline fails if any command in the chain fails.

Together, these three options catch the vast majority of scripting bugs at the point of failure rather than letting them cascade into more confusing symptoms downstream. ShellCheck is a static analysis tool that complements these runtime options by catching quoting errors, undefined variable references, and other common mistakes before the script runs at all; running it on any script before deploying is a reliable way to surface the class of bugs that set -euo pipefail cannot detect.

Cleanup with trap. Sometimes a script creates temporary files or acquires locks that must be released even if the script fails partway through. The trap builtin registers a command to run when the script receives a signal or exits:

#!/usr/bin/env bash
set -euo pipefail

TMPDIR=$(mktemp -d)
trap 'rm -rf "$TMPDIR"' EXIT

cp -r /var/www/html "$TMPDIR/html-staging"
tar czf /backups/html-latest.tar.gz -C "$TMPDIR" html-staging

echo "Backup archived successfully"

The trap ... EXIT fires regardless of whether the script succeeds or fails, so the temporary directory is always cleaned up. This is the Bash equivalent of a finally block in other languages.

The following script brings all the concepts together: safe defaults, a logging function, per-directory backup with error counting, and rotation:

#!/usr/bin/env bash
set -euo pipefail

BACKUP_ROOT="/backups"
TIMESTAMP=$(date +%Y%m%d-%H%M%S)
KEEP_DAYS=7
LOG="/var/log/backup.log"

log() {
    printf "[%s] %s\n" "$(date '+%Y-%m-%d %H:%M:%S')" "$1" | tee -a "$LOG"
}

backup_dir() {
    local source="$1"
    local dest="$BACKUP_ROOT/$(basename "$source")-$TIMESTAMP"
    if [[ ! -d "$source" ]]; then
        log "WARN: $source does not exist, skipping"; return 1
    fi
    cp -r "$source" "$dest"
    log "OK: backed up $source to $dest"
}

rotate_old() {
    find "$BACKUP_ROOT" -mindepth 1 -maxdepth 1 -type d -mtime +"$KEEP_DAYS" -exec rm -rf {} +
}

log "=== Backup run starting ==="
DIRS=("/var/www/html" "/etc/nginx" "/etc/ssh")
FAILURES=0

for dir in "${DIRS[@]}"; do
    if ! backup_dir "$dir"; then
        FAILURES=$(( FAILURES + 1 ))
    fi
done

rotate_old

if [[ "$FAILURES" -gt 0 ]]; then
    log "Completed with $FAILURES warning(s)"; exit 1
else
    log "All backups completed successfully"
fi

Regular Expressions

Regular expressions (universally shortened to regex) are a pattern language for text. A regex describes a set of strings: any line containing an IP address, any filename ending in .log, any timestamp in a particular format. The same core syntax is shared by grep, sed, awk, vi, and most general-purpose programming languages including Python, JavaScript, and Ruby. Learning regex once pays dividends across every text-processing tool you will encounter. Barnett’s Introduction to Regular Expressions at the Grymoire is a thorough reference for the full syntax and the subtle differences between tools.

The most important distinction in practice is between basic regular expressions (BRE), the default for grep, and extended regular expressions (ERE), used by grep -E. BRE requires backslash-escaping for operators like +, ?, |, and {n,m} to activate them as metacharacters; ERE treats them as metacharacters directly and requires no escaping. ERE syntax is more readable, and many modern regex engines use a similar notation, so this lecture uses it throughout.

Basic Operators

The following operators form the core of any regular expression. In ERE syntax, none of them require backslash-escaping to activate:

Operator	Meaning
`.`	Matches any single character
`*`	Matches zero or more occurrences of the previous character or group (the Kleene star)
`+`	Matches one or more occurrences
`?`	Matches zero or one occurrence
`^`	Anchors the match to the beginning of the line
`$`	Anchors the match to the end of the line
`\`	Escapes the next character so it is treated literally

The * in a regex is different from the * wildcard in shell globbing. In a regex, A* means “zero or more A characters.” In the shell, *.log means “any filename ending in .log.” The two syntaxes look similar but describe entirely different operations.

Character Classes

Square brackets match any one character from a set. The class can enumerate specific characters, negate them with ^, or specify a range:

Pattern	Matches
`[abc]`	Any one of `a`, `b`, or `c`
`[^abc]`	Any character except `a`, `b`, or `c`
`[a-z]`	Lowercase ASCII letters in the C/POSIX locale
`[0-9]`	Any digit

Character ranges are locale-sensitive in POSIX tools. If you need locale-aware alphabetic matching, POSIX character classes like [[:lower:]] and [[:digit:]] are usually safer than hard-coded ranges.

Anchoring and Exact Matching

Without anchors, a regex matches anywhere in the line. Anchors constrain where the match can occur:

Pattern	Behaviour
`Jon`	Matches any line containing `Jon`
`^Jon`	Matches lines that begin with `Jon`
`Jon$`	Matches lines that end with `Jon`
`^CS312$`	Matches only the exact string `CS312`

Repetition with Curly Braces

Curly braces specify how many times the preceding element must appear. In ERE syntax, the braces do not need to be escaped:

Pattern	Meaning
`{3}`	Exactly 3 times
`{3,7}`	Between 3 and 7 times (inclusive)
`{3,}`	At least 3 times

Grouping, Alternation, and Backreferences

Parentheses serve two purposes in a regular expression: they group sub-expressions for repetition or alternation, and they capture the matched text for reuse as a backreference. The | operator matches either of two alternatives. In ERE syntax (grep -E), parentheses and | work directly; in BRE syntax (plain grep), they must be backslash-escaped:

grep -E "cat|dog" file
grep -E "i like (cat|dog)" file

A backreference (\1, \2, etc.) refers back to the text matched by the first or second captured group. In BRE syntax, this matches any line where the word “dogs” appears twice in a row:

grep "\(dogs\) \1" file

Using `grep` for Filtering

grep (Global Regular Expression Print) is the primary tool for searching files and command output with regular expressions. It is most often used to filter lines from logs, command output, or configuration files. grep -E activates ERE syntax, avoiding backslash-escaping for |, +, and {n}. grep -i performs case-insensitive matching. grep -v inverts the match, printing only lines that do not contain the pattern. grep -rn searches recursively through a directory and prints line numbers alongside matches. grep -c counts matching lines instead of printing them.

grep -E "([0-9]{1,3}\.){3}[0-9]{1,3}" access.log

This ERE pattern matches lines containing an IPv4-like dotted quad. It is useful for quick log filtering, but it is not full IPv4 validation: it will also match values like 999.999.999.999 or a dotted quad embedded inside other text. The group [0-9]{1,3} matches one to three digits; \. matches a literal dot; {3} repeats the group three times for the first three octets; and the final [0-9]{1,3} matches the last octet.

`sed` for Stream Editing

sed (stream editor) applies regex-based substitutions to text one line at a time. Its core operation is the substitution command s/pattern/replacement/, with an optional g flag to replace all occurrences on a line rather than just the first:

sed 's/foo/bar/' input.txt
sed 's/foo/bar/g' input.txt
sed '/^#/d' config.txt
sed -i.bak 's/oldvalue/newvalue/g' config.txt

The third command deletes all comment lines from a config file (lines starting with #). The -i.bak form edits the file in place and keeps a backup as config.txt.bak; this works on both GNU sed and BSD/macOS sed. Without -i, the file is never changed and the command outputs to the terminal. sed is the right tool for line-by-line text transformation; it is less suited to extracting structured data from tabular output, which is where awk fits better.

`awk` for Field Processing

awk (named for its creators Aho, Weinberger, and Kernighan) is a field-oriented text processor. Where sed applies substitutions to whole lines, awk treats each line as a sequence of whitespace-separated fields and lets you write programs that operate on individual fields. The fields in each line are accessible as $1, $2, $3, and so on, with $0 representing the entire line.

This makes awk the natural tool for tabular output: command output from ps, df, or ls -l, log files with fixed column positions, or any delimiter-separated data. The following examples cover the most common operational patterns:

df -h | awk 'NR > 1 {print $6, $5}'
awk '$3 > 80 {print $1, $3}' cpu-log.txt
awk '{sum += $5} END {print "Total:", sum, "bytes"}' sizes.txt

The first prints the mount point and usage percentage for every mounted filesystem, skipping the header row (NR > 1 means “line number greater than 1”). The second prints the hostname and CPU percentage for any line where the third field exceeds 80. The third sums a column across all input lines and prints the result once, using the END block that runs after all input is consumed.

The conceptual distinction between sed and awk is this: reach for sed when you need to find and replace text within lines, and reach for awk when you need to work with specific columns or aggregate values across rows. When an awk one-liner grows into multiple rules or requires custom functions, Python handles the same task more readably. Python’s re module (re.findall(), re.sub(), re.search()) supports the same ERE-style patterns covered here, using raw strings (r'\d{3}') to avoid double-escaping backslashes. When text processing outgrows a pipeline and needs real data structures or control flow, the regex knowledge transfers directly.

Scheduling with Cron

A backup script is only useful if it runs on a schedule. Cron is the traditional Unix job scheduler and is installed on most general-purpose Linux systems, though some minimal containers and cloud images omit the daemon entirely. Each user has a crontab (cron table) that lists commands and their schedules. Edit yours with crontab -e. If that command is missing or cron is not running, the system may not have cron installed. The format is five fields followed by the command:

0    2     *    *      *         /usr/local/bin/backup.sh

The five fields are minute (0-59), hour (0-23), day of month (1-31), month (1-12), and day of week (0-7, where both 0 and 7 mean Sunday). This entry runs backup.sh every day at 2:00 AM. The @daily, @hourly, and @weekly shortcuts are supported by most cron implementations and are easier to read than the five-field syntax.

Cron runs with a minimal environment. The PATH it provides typically contains only /usr/bin:/bin, which means commands that work in your interactive shell may fail under cron with “command not found.” Always use absolute paths for your script and every tool it calls. Redirect both stdout and stderr to a log file with >> /var/log/backup-cron.log 2>&1 so that failures leave evidence you can diagnose later. Before deploying a cron schedule, verify it at crontab.guru, which translates any five-field expression into plain English.

On modern Linux distributions, systemd timers are a more capable alternative: they integrate with journalctl, support a Persistent option that fires missed runs at the next boot rather than silently skipping them, and allow CPU and memory limits to be applied to the scheduled task. The Linux Server Planning and Configuration lecture covers the unit file structure in detail. For most single-machine scripting contexts, cron remains the simpler and more universally available choice.

When to Stop Scripting and Use a Real Tool

Bash is an excellent tool for automating tasks on a single machine, but it has limits. As your scripts grow in complexity, watch for these warning signs.

You are managing multiple hosts. A Bash script that SSHs into a dozen servers in a loop is brittle. Connection failures, partial runs, and inconsistent state are hard to handle. Configuration management tools like Ansible were designed for exactly this problem. Ansible is agentless: it connects to managed nodes over SSH without requiring a persistent daemon or agent on each server. It adds idempotence (the property that running a command multiple times produces the same result as running it once, with no unintended side effects), inventory management, and error handling that would take hundreds of lines of Bash to replicate.

You are parsing structured data. Bash can manipulate strings, but parsing JSON, YAML, or XML in Bash is painful and error-prone. Python, with libraries like json, pyyaml, and requests, handles structured data naturally and provides real data structures and exception-based error handling that make complex transformations readable and safe in ways Bash cannot match. jq (a command-line JSON processor) is also worth knowing for one-off JSON queries in pipelines without leaving the shell entirely. If your script has more than a couple of calls to jq or awk for structured data, consider whether the task belongs in Python instead.

Your script exceeds a few hundred lines. Bash has no real module system, limited error handling, and no type safety. Once a script becomes long enough that you need to scroll to understand it, the maintenance cost exceeds the benefit of staying in Bash. Python’s module system, clear function signatures, and readable syntax make larger programs navigable in ways Bash cannot.

You need testability. Writing automated tests for Bash scripts is possible but awkward. Python’s pytest is a mature framework that makes it straightforward to verify behavior and catch regressions.

The practical rule is this: start with Bash for simple, single-machine automation. When the task outgrows Bash’s strengths, move to the right tool for the job. That might be Ansible for multi-host configuration, Python for data processing, or Terraform for infrastructure provisioning. The scripting fundamentals covered in this lecture transfer directly to those tools, because they all build on the same Unix concepts.

Scripting Languages in Context

Bash and Python are the two languages a Linux administrator is most likely to write. Beyond those, the scripting landscape is wide, and different environments favor different tools. Understanding the rough position of each language helps you read job descriptions, understand what tools are written in, and choose the right language when the task does not fit neatly into Bash or Python.

PowerShell is the native scripting language for Windows Server and, since PowerShell Core (2018), runs cross-platform on Linux and macOS as well. Its most significant architectural difference from Bash is that it passes objects between pipeline stages rather than text: a Get-Process | Where-Object {$_.CPU -gt 10} pipeline filters structured process objects, not formatted output lines. Microsoft’s management surfaces for Active Directory and Exchange remain especially PowerShell-centric, and Azure has first-class PowerShell support alongside Azure CLI, REST APIs, and SDKs. In mixed Linux and Windows environments, PowerShell Core is increasingly used as a common scripting layer across both.

Go is the language of choice for building system tools that are distributed as compiled binaries. Docker, Kubernetes, Terraform, and kubectl are all written in Go. It compiles to a single static binary with no runtime dependency, which simplifies deployment significantly compared to Python or Ruby. When an automation script evolves into a tool that other people will install and run, Go’s deployment model and static typing make the higher entry cost worthwhile.

Ruby was the dominant infrastructure automation language from roughly 2008 to 2015, primarily through the Chef and Puppet configuration management systems. Its prevalence has declined as Ansible (which uses YAML rather than a programming language) displaced Chef and Puppet, and as Python became the general-purpose scripting default. Legacy Ruby automation is still common in some enterprise environments.

Perl was the dominant sysadmin scripting language before Python matured. It shipped on every Unix system, had unmatched regex support, and was the standard choice for CGI web scripting and log processing through the 1990s and early 2000s. New Perl code is rare, but reading and modifying existing Perl scripts remains a practical skill in environments built during that era.

JavaScript and Node.js appear most often in automation for web-adjacent organizations: CI/CD pipelines, serverless functions, and tooling in JavaScript-heavy teams. The npm ecosystem is large, but the toolchain overhead is a poor fit for straightforward system scripts.

SQL is not a scripting language in the traditional sense, but it is the right tool when the task is fundamentally a data query rather than a sequence of system operations. Aggregating log records, reporting on server inventory, or summarizing metrics stored in a relational database is best expressed in SQL, where a few lines replace pages of Bash or Python that try to replicate set operations with loops.

Takeaways

Shell scripting occupies a specific position in the automation landscape. The Infrastructure as Code lecture introduced the declarative alternative: describe the desired end state and let the tool figure out how to reach it. Shell scripts are the opposite approach, imperative and sequential, encoding exactly the steps to perform a task. That imperative approach is the right one for single-machine automation where the sequence matters and where you are not managing idempotency at scale. Bash requires nothing beyond the shell already on the server and produces output you can read and log.

The building blocks of this lecture connect in a realistic backup script: variables and quoting prevent path-splitting bugs, script arguments let the same logic run against any source and destination, the exported environment controls what child processes can see, conditionals validate preconditions before any work begins, loops process arrays of directories, functions isolate reusable logic, and set -euo pipefail catches undefined variables, broken pipelines, and unhandled command failures near the point where they occur. Exit codes carry success or failure to cron, systemd, and any calling script. A trap EXIT clause ensures cleanup runs regardless of how the script exits. Scheduling the script with cron automates the work; the minimal cron environment means absolute paths are essential, and appending both stdout and stderr to a log file with >> /var/log/backup.log 2>&1 ensures failures leave evidence that can be diagnosed later.

Regular expressions extend Bash’s reach into text processing. The pattern language is the same across grep, sed, awk, and most programming languages; learning it once applies everywhere. The distinction to carry forward is what each tool is best at: grep filters lines, sed transforms lines with substitutions, and awk processes fields within lines and can aggregate values across rows.

Bash is the right tool until it is not. The practical threshold arrives at one of three conditions: multiple hosts (the Configuration Management with Ansible lecture describes the right tool for fleet-scale automation), structured data (Python handles JSON and YAML naturally), or scripts long enough to require scrolling. The underlying primitives do not change when you switch tools. SSH, environment variables, exit codes, stdin, stdout, and stderr are the vocabulary of Unix automation regardless of whether you are writing Bash, Python, or an Ansible playbook. Ansible connects to managed nodes over SSH, reads configuration from environment variables, and determines task success or failure from exit codes. Every concept from this lecture transfers directly.