Text Processing, Pipes & Redirection

Almost everything a DevOps junior debugs shows up as text — a log file, a command's output, a config file. The shell's real power isn't any single tool; it's that small, single-purpose programs can be chained together so each one's output becomes the next one's input, turning a few simple commands into a precise query over a mountain of text.

TL;DR — in 30 seconds:

• A pipe (|) sends one command's stdout straight into the next command's stdin — no temporary file needed.
• grep finds lines, sed transforms them, awk does column-aware processing and computation — reaching for the wrong one is the most common beginner mistake.
• Redirection (>, >>, <, 2>) moves a stream to or from a file instead of the terminal; a pipe moves a stream to or from another command — different targets, same underlying idea.

1. Streams, pipes, and redirection

Every process has three standard streams: stdin (input, fd 0), stdout (normal output, fd 1), and stderr (error output, fd 2, kept separate from stdout on purpose so errors don't get mixed into data you're processing).

flowchart LR
    A["cat access.log"] -->|stdout piped to stdin| B["grep ERROR"]
    B -->|stdout piped to stdin| C["awk '{print $1}'"]
    C -->|stdout piped to stdin| D["sort"]
    D -->|stdout piped to stdin| E["uniq -c"]

• Pipe (|) — connects one command's stdout to the next command's stdin, in memory, no file written.
• Redirection sends a stream to/from a file instead:

Operator	Effect
>	redirect stdout to a file, overwriting it
>>	redirect stdout to a file, appending to it
<	feed a file's contents in as stdin
2>	redirect stderr only, to a file
&> (or > file 2>&1)	redirect both stdout and stderr to the same file

Redirection and pipes compose: command < input.txt | grep foo > output.txt reads from a file, filters it, and writes the result to another file.

2. grep — finding lines

grep <pattern> <file> prints every line matching a pattern (a plain string or a regular expression).

Flag	Meaning
-i	case-insensitive match
-r	recurse into a directory
-v	invert — print lines that do not match
-c	print a count of matching lines instead of the lines themselves
-n	prefix each match with its line number
-E	extended regular expressions (so +, ?, \| work unescaped)

grep answers "which lines match" — it never changes or rearranges anything.

3. sed — transforming lines

sed (stream editor) applies an edit to every line as it streams through. The workhorse is the substitute command: sed 's/old/new/' file replaces the first match of old with new on each line; adding a g flag (s/old/new/g) replaces every match on the line, not just the first.

• sed -i 's/old/new/' file — edit the file in place (no output to the terminal; the file itself changes). Without -i, sed only prints the transformed result — the source file is untouched.
• sed can also delete lines matching a pattern (sed '/pattern/d') or print only a range of lines.

sed answers "what should each line become" — a targeted find-and-replace across a stream.

4. awk — column-aware processing

awk splits each input line into whitespace-separated fields — $1 is the first field, $2 the second, and so on, $0 is the whole line — and runs a pattern/action program per line.

• awk '{print $1}' file — print just the first column of every line.
• awk -F, '{print $2}' file — same, but split on , instead of whitespace (-F sets the field separator).
• awk '$3 > 100 {print $1}' file — a condition before the {action}: only run the action on lines matching it (here, third field greater than 100).
• awk can also aggregate — awk '{sum += $2} END {print sum}' sums a column and prints the total once, at the end of the input.

awk answers "compute something from this line's fields" — the tool for column math and conditional extraction that grep/sed can't do.

5. Rounding out a pipeline: cut, sort, uniq

• cut — extract a slice without awk's full field-processing: cut -d',' -f2 (delimiter ,, field 2) or cut -c1-10 (characters 1 through 10).
• sort — sort lines: alphabetically by default, -n for numeric order, -r to reverse, -k2 to sort by the 2nd field.
• uniq — collapse adjacent duplicate lines (it does not look at the whole file — always sort first if the duplicates aren't already next to each other). uniq -c prefixes each line with how many times it repeated.

A very common pipeline shape — "which values appear most often" — is exactly the diagram above: grep to filter down to the relevant lines, awk to pull out the field you care about, sort to bring duplicates together, uniq -c to count them.

Common gotchas

• Reaching for awk (or a script) when grep/cut already does it. Fix: match the tool to the job — "which lines" is grep, "which columns" is cut, "transform each line" is sed, and only reach for awk when you need conditions or computation across fields.
• Running uniq without sort first and getting duplicates that "don't count." Fix: uniq only collapses adjacent identical lines — sort the input first so every duplicate is next to its copies.
• Forgetting sed's -i writes the file with no confirmation. Fix: run without -i first to preview the transformed output, then add -i once you've confirmed it's correct — there's no undo.
• Mixing up > and >> and clobbering a file you meant to append to. Fix: > truncates the file first, every time; >> appends. If a log file mysteriously "lost its history," check whether the command that wrote to it used > where it needed >>.

Check yourself — you have a log file and want the 10 most frequent values in its 3rd column. Which commands do you chain, and in what order?

awk '{print $3}' file | sort | uniq -c | sort -rn | head -10 — pull the 3rd field, sort so identical values are adjacent, count adjacent duplicates with uniq -c, sort the counts numerically descending, then take the top 10.

Check yourself — grep ERROR app.log > errors.txt runs twice in a row. What's in errors.txt after the second run, and why?

Only the results of the second run — > overwrites (truncates) the file every time it's used, so the first run's output is gone. Using >> on the second run would have appended instead, keeping both.

6. How this connects

• This course's next page, 03-process-monitoring.md — a ps aux snapshot is exactly the kind of text stream this page's tools filter: ps aux | grep <name> finds one process by name, and awk '{print $2}' pulls just its PID column out of the result.
• CI/CD (module 15, CI/CD, the run: step) — a workflow's run: step is a raw shell command, and these are the tools that fill it: a build script that greps a command's output for a pass/fail marker, or pipes a test report through awk/sort to summarize it, is the same pipeline shape taught here, just running on a CI runner instead of your terminal.
• Observability (module 17, Observability & Monitoring, the logs pillar) — that module's "logs tell you why, one discrete event at a time" is exactly what greping a raw log file does by hand: before a centralized tool like Grafana exists to query them, grep/awk over a log file is the logs pillar in practice.

You can defend this when you can pick the right tool (grep/sed/awk/cut/sort/uniq) for a given text-processing task without reaching for the heaviest one out of habit, and can explain the difference between a pipe and a redirect.