A comprehensive guide to every major Linux text processing tool: grep, sed, awk, tr, cut, sort, uniq, wc, paste, xargs, and tee. Covers syntax, flags, real examples, pipelines, and sysadmin/pentesting use cases.
1. Overview — Which Tool Does What?
TOOL PURPOSE COMPLEXITY
──────────────────────────────────────────────────────────────────
grep Search / filter lines by pattern Low — simple search
sed Find + replace / delete / insert Medium — regex substitution
awk Field-based processing + logic High — mini programming language
tr Translate/delete single characters Low — character-level only
cut Extract columns/fields Low — positional extraction
sort Sort lines Low — ordering
uniq Remove/count duplicate lines Low — deduplication
wc Count lines, words, characters Low — counting
head Show first N lines Low — preview
tail Show last N lines Low — monitoring
paste Merge lines/files side-by-side Low — joining
tee Write to file AND stdout Low — branching output
xargs Build commands from stdin input Medium — execution bridge
WHEN TO USE WHICH:
Need to FIND lines? → grep
Need to REPLACE text? → sed
Need to PROCESS columns? → awk or cut
Need to CHANGE chars? → tr
Need to SORT output? → sort
Need to DEDUPLICATE? → sort | uniq
Need to COUNT things? → wc or awk
Need to BUILD commands? → xargs
UNIX PHILOSOPHY — combine small tools with pipes (|):
cat file | grep 'ERROR' | awk '{print $1}' | sort | uniq -c | sort -rn
^^read ^^filter ^^extract field ^^sort ^^deduplicate ^^rank
2. grep — Search & Filter Text
# ==========================================
# GREP — Global Regular Expression Print
# ==========================================
# Reads input line by line and prints lines that MATCH a pattern.
# SYNTAX:
# grep [OPTIONS] 'PATTERN' file(s)
# command | grep [OPTIONS] 'PATTERN'
# THREE MODES:
grep 'pattern' file # BRE — Basic Regex (default)
grep -E 'pattern' file # ERE — Extended Regex (cleaner syntax)
grep -P 'pattern' file # PCRE — Perl Compatible (most powerful)
# ──────────────────────────────────────────
# ALL FLAGS — EXPLAINED
# ──────────────────────────────────────────
grep -i 'pattern' file # Case INSENSITIVE
grep -v 'pattern' file # INVERT — print lines NOT matching
grep -n 'pattern' file # Show LINE NUMBERS
grep -c 'pattern' file # COUNT matching lines only
grep -l 'pattern' files # List FILENAMES that contain match
grep -L 'pattern' files # List filenames that DON'T match
grep -o 'pattern' file # Print ONLY matched text (not full line)
grep -w 'pattern' file # Match WHOLE WORD only
grep -x 'pattern' file # Match WHOLE LINE only
grep -r 'pattern' dir/ # RECURSIVE search
grep -R 'pattern' dir/ # Recursive + follow symlinks
grep -A 3 'pattern' file # 3 lines AFTER match
grep -B 3 'pattern' file # 3 lines BEFORE match
grep -C 3 'pattern' file # 3 lines before AND after
grep -m 5 'pattern' file # Stop after 5 matches
grep -q 'pattern' file # Quiet — only exit code (0=found)
grep -h 'pattern' files # No filename prefix in output
grep -s 'pattern' file # Suppress file error messages
grep -f patterns.txt file # Read patterns from file
grep --color 'pattern' file # Highlight matched text in color
grep --include="*.py" -r 'def ' ./ # Search only .py files
grep --exclude="*.log" -r 'error' ./ # Exclude .log files
grep --exclude-dir={.git,node_modules} -r 'TODO' ./
# ──────────────────────────────────────────
# BASIC EXAMPLES
# ──────────────────────────────────────────
grep 'root' /etc/passwd # Lines containing "root"
grep '^root' /etc/passwd # Lines STARTING with "root"
grep 'bash$' /etc/passwd # Lines ENDING with "bash"
grep '^root.*bash$' /etc/passwd # Starts with root, ends with bash
grep '^$' file # Empty lines only
grep -v '^$' file # Non-empty lines
grep -v '^#' /etc/hosts # Remove comment lines
grep -c 'error' /var/log/syslog # Count error lines
grep -n 'fail' /var/log/auth.log # Show line numbers
# ──────────────────────────────────────────
# ERE EXAMPLES (grep -E)
# ──────────────────────────────────────────
grep -E 'error|warning|critical' syslog # OR — multiple patterns
grep -E '^(root|daemon|nobody):' /etc/passwd # Multiple users
grep -E '[0-9]{4}-[0-9]{2}-[0-9]{2}' file # Date YYYY-MM-DD
grep -E '[0-9]{1,3}(\.[0-9]{1,3}){3}' file # IPv4 address
grep -E '^.{80,}$' file # Lines 80+ chars long
grep -E '(https?|ftp)://' file # URLs (http or https or ftp)
grep -E '([a-z]+)\s+\1' file # Doubled words
# ──────────────────────────────────────────
# PCRE EXAMPLES (grep -P) — MOST POWERFUL
# ──────────────────────────────────────────
grep -P '\d+' file # Lines with digits (\d = PCRE only)
grep -P '\b\w{10,}\b' file # Words 10+ chars long
grep -P '(?<=password: )\S+' file # Lookbehind — text after "password: "
grep -P '^\s*$' file # Blank/whitespace-only lines
grep -P '\t' file # Lines with TAB character
grep -P '(?i)error' file # Case insensitive via inline flag
# ──────────────────────────────────────────
# GREP IN PIPELINES
# ──────────────────────────────────────────
ps aux | grep nginx | grep -v grep # Find nginx processes
dmesg | grep -iE 'error|fail|warn' # Kernel errors
history | grep -E 'ssh|curl|wget' # Command history filter
ss -tlnp | grep -E ':(80|443|22)\s' # Specific ports
journalctl | grep -P 'Failed|Invalid' # Auth failures
# Extract all unique IPs from log:
grep -oP '\b\d{1,3}(\.\d{1,3}){3}\b' access.log | sort -u
# Find files containing string recursively:
grep -rl 'TODO' /home/zoro/project/ --include="*.py"
# Count HTTP status codes:
grep -oE '"[0-9]{3}"' access.log | sort | uniq -c | sort -rn
3. sed — Stream Editor
# ==========================================
# SED — Stream Editor
# ==========================================
# Reads file line by line, applies commands, outputs result.
# NON-INTERACTIVE — changes flow through like a stream.
# SYNTAX:
# sed [OPTIONS] 'COMMAND' file
# sed [OPTIONS] -e 'CMD1' -e 'CMD2' file (multiple commands)
# sed [OPTIONS] 's/FIND/REPLACE/FLAGS' file (substitution)
# KEY OPTIONS:
# -n Suppress automatic print (use p to print manually)
# -E Use ERE (cleaner + ? | ( ) without backslash)
# -i Edit file IN-PLACE
# -i.bak Edit in-place with .bak backup
# -e Add multiple commands
# -f Read commands from a script file
# ──────────────────────────────────────────
# SUBSTITUTION — THE MOST USED COMMAND
# ──────────────────────────────────────────
# s/FIND/REPLACE/FLAGS
# The delimiter / can be ANY character — useful for paths!
# s|/old/path|/new/path|g ← use | to avoid escaping slashes
# s#old#new#g ← use # as delimiter
sed 's/old/new/' file # Replace FIRST occurrence per line
sed 's/old/new/g' file # Replace ALL occurrences (global)
sed 's/old/new/2' file # Replace 2nd occurrence only
sed 's/old/new/3g' file # Replace from 3rd occurrence onwards
sed 's/old/new/gi' file # Replace all, CASE INSENSITIVE
sed 's/old/new/gp' file # Replace all + print changed lines
# & = the entire matched text (placeholder in replacement):
sed 's/[0-9]+/[&]/g' file # Input: "port 8080" → "port "
sed 's/.*/--- & ---/' file # Wrap every line: --- line content ---
# ──────────────────────────────────────────
# BRE vs ERE IN SED
# ──────────────────────────────────────────
# BRE (default) — quantifiers need backslash:
sed 's/[0-9]\+/NUM/g' file # \+ = one or more (BRE)
sed 's/[0-9]\{4\}/YEAR/g' file # \{4\} = exactly 4 (BRE)
# ERE (sed -E) — cleaner:
sed -E 's/[0-9]+/NUM/g' file # + without backslash!
sed -E 's/[0-9]{4}/YEAR/g' file # {} without backslash!
# ──────────────────────────────────────────
# BACKREFERENCES — CAPTURE AND REUSE
# ──────────────────────────────────────────
# \1 \2 refer to capture groups in replacement
# Swap first and second words:
sed -E 's/(\w+) (\w+)/\2 \1/' file
# Input: "hello world"
# Output: "world hello"
# Reformat date MM/DD/YYYY → YYYY-MM-DD:
sed -E 's|([0-9]{2})/([0-9]{2})/([0-9]{4})|\3-\1-\2|g' file
# Input: "03/15/2024"
# Output: "2024-03-15"
# Extract username from /etc/passwd:
sed -E 's/([^:]+):.*/\1/' /etc/passwd
# Input: "root:x:0:0:root:/root:/bin/bash"
# Output: "root"
# Wrap numbers in brackets:
sed -E 's/([0-9]+)/[\1]/g' file
# Input: "Port 443 and 8080"
# Output: "Port and "[11]
# Add quotes around each word:
sed -E 's/(\w+)/"\1"/g' file
# ──────────────────────────────────────────
# LINE ADDRESSING — TARGET SPECIFIC LINES
# ──────────────────────────────────────────
sed '5s/old/new/' file # Only line 5
sed '1,10s/old/new/g' file # Lines 1 to 10
sed '5,$ s/old/new/g' file # Line 5 to end of file
sed '1~2s/old/new/g' file # Every 2nd line (1,3,5...)
sed '/pattern/s/old/new/g' file # Lines matching pattern
sed '/pat/!s/old/new/g' file # Lines NOT matching pattern
sed '/start/,/end/s/old/new/g' file # Between two patterns
# ──────────────────────────────────────────
# OTHER SED COMMANDS
# ──────────────────────────────────────────
# d — DELETE lines:
sed '/pattern/d' file # Delete lines matching pattern
sed '/^$/d' file # Delete empty lines
sed '/^#/d' file # Delete comment lines
sed '5d' file # Delete line 5
sed '1,3d' file # Delete lines 1-3
# p — PRINT lines (used with -n):
sed -n '/pattern/p' file # Print ONLY matching lines (like grep!)
sed -n '5,10p' file # Print lines 5 to 10
sed -n '$p' file # Print last line only
# i — INSERT before line:
sed '/pattern/i\NEW LINE BEFORE' file
sed '1i\HEADER' file # Insert header at top
# a — APPEND after line:
sed '/pattern/a\NEW LINE AFTER' file
sed '$a\FOOTER' file # Append footer at end
# c — CHANGE/REPLACE entire line:
sed '/pattern/c\REPLACEMENT LINE' file
# q — QUIT after N lines:
sed '10q' file # Print first 10 lines then quit (like head -10)
# = — Print line number:
sed -n '/pattern/=' file # Print line numbers of matches
# y — TRANSLITERATE (like tr):
sed 'y/abc/ABC/' file # Replace a→A, b→B, c→C
# ──────────────────────────────────────────
# IN-PLACE EDITING
# ──────────────────────────────────────────
sed -i 's/foo/bar/g' file # Edit file in-place!
sed -i.bak 's/foo/bar/g' file # With .bak backup
sed -i 's/http:/https:/g' *.conf # Multiple files!
sed -i '/^#/d; /^$/d' config.txt # Multiple commands
# ──────────────────────────────────────────
# PRACTICAL SED EXAMPLES
# ──────────────────────────────────────────
# Remove trailing whitespace:
sed -E 's/[[:space:]]+$//' file.txt
# Remove leading whitespace:
sed -E 's/^[[:space:]]*//' file.txt
# Remove both:
sed -E 's/^[[:space:]]*|[[:space:]]*$//g' file.txt
# Remove HTML tags:
sed -E 's/<[^>]+>//g' page.html
# Remove ANSI color codes:
sed -E 's/\x1B\[[0-9;]*[mGKHF]//g' output.txt
# Fix Windows CRLF → Unix LF:
sed -i 's/\r//' file.txt
# Comment out a line matching pattern:
sed -E 's/(^PermitRootLogin.*)/#\1/' /etc/ssh/sshd_config
# Double-space a file (blank line after each line):
sed 'G' file.txt
# Print lines between two patterns:
sed -n '/START/,/END/p' file.txt
# Multiple operations at once:
sed -e 's/foo/bar/g' -e 's/baz/qux/g' file.txt
sed 's/foo/bar/g; s/baz/qux/g' file.txt # Same thing
# Extract IPs from file:
sed -nE 's/.*([0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}).*/\1/p' file
4. awk — Field Processor & Mini Language
# ==========================================
# AWK — Aho, Weinberger, Kernighan
# ==========================================
# A complete programming language for text processing.
# Splits each line into FIELDS and lets you process them with logic.
# SYNTAX:
# awk 'BEGIN{setup} /pattern/{action} END{teardown}' file
# awk [OPTIONS] 'program' file(s)
# KEY CONCEPTS:
# $0 = entire current line
# $1 = first field
# $2 = second field
# $NF = LAST field (NF = number of fields)
# $(NF-1) = second-to-last field
# NR = current line number (across all files)
# FNR = line number within current file
# NF = number of fields on current line
# FS = field separator (default: whitespace)
# OFS = output field separator (default: space)
# RS = record separator (default: newline)
# ORS = output record separator (default: newline)
# FILENAME = current filename being processed
# ──────────────────────────────────────────
# FIELD PRINTING — MOST BASIC USE
# ──────────────────────────────────────────
awk '{print $1}' file # Print first field
awk '{print $2}' file # Print second field
awk '{print $NF}' file # Print last field
awk '{print $1, $3}' file # Print fields 1 and 3
awk '{print $0}' file # Print entire line (same as cat)
awk '{print NR, $0}' file # Print with line numbers
awk '{print NF}' file # Print number of fields per line
# With custom field separator (-F):
awk -F: '{print $1}' /etc/passwd # Colon-separated → print username
awk -F, '{print $2}' data.csv # CSV → print 2nd column
awk -F'\t' '{print $3}' file.tsv # Tab-separated
awk -F'[,;|]' '{print $1}' file # Multiple delimiters (regex!)
awk 'BEGIN{FS=":"} {print $1, $3}' /etc/passwd # Set FS in BEGIN
# ──────────────────────────────────────────
# PATTERN MATCHING
# ──────────────────────────────────────────
awk '/error/' file # Print lines matching "error"
awk '/^[0-9]/' file # Lines starting with digit
awk '!/^#/' file # Lines NOT starting with #
awk '/start/,/end/' file # Lines between two patterns
# Match on specific field:
awk '$1 ~ /192\.168/' file # Field 1 matches pattern (~)
awk '$3 ~ /[0-9]+/' file # Field 3 contains digits
awk '$2 !~ /skip/' file # Field 2 does NOT match
awk '$3 == "ERROR"' file # Field 3 equals exactly "ERROR"
awk '$3 != "INFO"' file # Field 3 NOT equal to INFO
awk '$5 > 100' file # Field 5 is greater than 100
awk 'NF > 3' file # Lines with more than 3 fields
awk 'NR==5' file # Only line 5
awk 'NR>=5 && NR<=10' file # Lines 5-10
# ──────────────────────────────────────────
# BEGIN AND END BLOCKS
# ──────────────────────────────────────────
# BEGIN runs BEFORE processing any lines:
awk 'BEGIN{print "=== Report ==="} {print $1}' file
# END runs AFTER all lines are processed:
awk 'END{print "Total lines:", NR}' file
# Full program with all three:
awk 'BEGIN{count=0}
/error/{count++}
END{print "Errors found:", count}' logfile
# ──────────────────────────────────────────
# ARITHMETIC AND VARIABLES
# ──────────────────────────────────────────
# Sum a column:
awk '{sum += $3} END{print "Total:", sum}' file
# Average:
awk '{sum += $3; count++} END{print "Avg:", sum/count}' file
# Min and Max:
awk 'BEGIN{max=0} $3>max{max=$3} END{print "Max:", max}' file
# Count lines matching pattern:
awk '/error/{count++} END{print count}' file
# Multiple counters:
awk '/ERROR/{e++} /WARNING/{w++} /INFO/{i++}
END{print "E:"e, "W:"w, "I:"i}' logfile
# ──────────────────────────────────────────
# CONDITIONALS AND LOOPS
# ──────────────────────────────────────────
# if/else:
awk '{if($3 > 200) print "HIGH:", $0; else print "LOW:", $0}' file
awk '{if($5 >= 400 && $5 < 500) print "4xx error:", $7}' access.log
# for loop:
awk '{for(i=1; i<=NF; i++) print i, $i}' file # Print each field on own line
# while loop:
awk '{i=1; while(i<=NF){print $i; i++}}' file
# ──────────────────────────────────────────
# ARRAYS (ASSOCIATIVE — HASH MAPS)
# ──────────────────────────────────────────
# Count occurrences of field 1:
awk '{count[$1]++} END{for(k in count) print count[k], k}' file
# Count IP hits in access log:
awk '{count[$1]++} END{for(ip in count) print count[ip], ip}' access.log | sort -rn
# Unique values of a field:
awk '!seen[$1]++{print $1}' file # Print unique values of field 1
# Group by field and sum:
awk -F, '{total[$1]+=$2} END{for(k in total) print k, total[k]}' data.csv
# ──────────────────────────────────────────
# STRING FUNCTIONS
# ──────────────────────────────────────────
awk '{print length($0)}' file # Length of each line
awk '{print toupper($1)}' file # Uppercase field 1
awk '{print tolower($2)}' file # Lowercase field 2
awk '{print substr($0, 1, 10)}' file # First 10 characters
awk '{print substr($1, 3)}' file # Field 1 from char 3 onwards
awk '{gsub(/foo/, "bar"); print}' file # Global replace in line
awk '{sub(/foo/, "bar"); print}' file # Replace first only
awk '{n=split($0,a,":"); print a[1], n}' file # Split and count
# index() — find position of substring:
awk '{pos=index($0, "error"); if(pos>0) print NR, pos}' file
# sprintf() — format strings:
awk '{printf "%-20s %10.2f\n", $1, $3}' file # Column-aligned output
awk '{print sprintf("%05d", NR), $0}' file # Zero-padded line numbers
# match() — find regex in string:
awk '{if(match($0, /[0-9]+/, a)) print a}' file
# ──────────────────────────────────────────
# OUTPUT FORMATTING
# ──────────────────────────────────────────
# print vs printf:
awk '{print $1, $2}' file # Space between, newline at end
awk '{print $1 "\t" $2}' file # Tab between fields
awk '{printf "%s,%s\n", $1, $2}' file # Explicit CSV output
awk '{printf "%-15s %d\n", $1, $3}' file # Left-aligned, 15 chars wide
# Custom Output Field Separator:
awk 'BEGIN{OFS=","} {print $1,$2,$3}' file # Output as CSV
awk 'BEGIN{OFS="\t"} {print $1,$2}' file # Output as TSV
# ──────────────────────────────────────────
# PRACTICAL AWK EXAMPLES
# ──────────────────────────────────────────
# /etc/passwd — username and shell:
awk -F: '{print $1, $7}' /etc/passwd
# /etc/passwd — users with bash:
awk -F: '$7 ~ /bash$/ {print $1}' /etc/passwd
# Apache log — top 10 IPs:
awk '{print $1}' access.log | sort | uniq -c | sort -rn | head -10
# Apache log — count by HTTP status:
awk '{print $9}' access.log | sort | uniq -c | sort -rn
# Apache log — requests over 1MB:
awk '$10 > 1048576 {print $7, $10}' access.log
# Sum file sizes from ls -l:
ls -l | awk '{sum += $5} END{print "Total:", sum, "bytes"}'
# CPU usage of top processes:
ps aux | awk 'NR>1 {sum += $3} END{print "Total CPU:", sum"%"}'
# Memory >1% usage:
ps aux | awk 'NR>1 && $4 > 1.0 {print $11, $4"%"}'
# Disk usage alert (>80%):
df -h | awk 'NR>1 && $5+0 > 80 {print "ALERT:", $6, $5}'
# Print every other line:
awk 'NR%2==0' file # Even lines
awk 'NR%2==1' file # Odd lines
# Reverse fields on each line:
awk '{for(i=NF;i>=1;i--) printf "%s%s",$i,(i>1?OFS:ORS)}' file
# Remove duplicate lines (preserving order):
awk '!seen[$0]++' file
# Transpose rows and columns:
awk '{for(i=1;i<=NF;i++) a[i][NR]=$i} END{for(i=1;i<=NF;i++){for(j=1;j<=NR;j++) printf "%s ",a[i][j]; print ""}}' file
5. tr — Translate Characters
# ==========================================
# TR — Translate or Delete Characters
# ==========================================
# Works on CHARACTER SETS — not full regex.
# No quantifiers (* + ? {}) or anchors (^ $).
# Always reads from stdin, writes to stdout.
# SYNTAX:
# tr [OPTIONS] SET1 [SET2]
# command | tr SET1 SET2
# OPTIONS:
# -d Delete characters in SET1 (no SET2 needed)
# -s Squeeze consecutive repeated chars in SET1 to one
# -c Complement — use chars NOT in SET1
# ──────────────────────────────────────────
# BASIC TRANSLATION
# ──────────────────────────────────────────
echo "hello" | tr 'a-z' 'A-Z' # → HELLO (lowercase → uppercase)
echo "HELLO" | tr 'A-Z' 'a-z' # → hello (uppercase → lowercase)
echo "hello world" | tr ' ' '_' # → hello_world (space → underscore)
echo "abc123" | tr '0-9' '#' # → abc### (digits → #)
echo "hello" | tr 'aeiou' '*' # → h*ll* (vowels → *)
# Using POSIX classes:
echo "Hello World" | tr '[:lower:]' '[:upper:]' # → HELLO WORLD
echo "HELLO WORLD" | tr '[:upper:]' '[:lower:]' # → hello world
echo "abc 123" | tr '[:digit:]' 'X' # → abc XXX
# ──────────────────────────────────────────
# DELETE CHARACTERS (-d)
# ──────────────────────────────────────────
echo "hello world" | tr -d ' ' # → helloworld (remove spaces)
echo "abc123" | tr -d '0-9' # → abc (remove digits)
echo "hello\r\n" | tr -d '\r' # → hello\n (remove CR)
echo "h-e-l-l-o" | tr -d '-' # → hello (remove hyphens)
cat file.txt | tr -d '[:punct:]' # Remove all punctuation
# ──────────────────────────────────────────
# SQUEEZE DUPLICATES (-s)
# ──────────────────────────────────────────
echo "hello world" | tr -s ' ' # → hello world (multiple spaces → one)
echo "aaabbbccc" | tr -s 'a-z' # → abc (squeeze all letters)
echo "1111222333" | tr -s '0-9' # → 123 (squeeze digits)
echo "hello world" | tr -s '[:space:]' ' ' # Normalize whitespace
# ──────────────────────────────────────────
# COMPLEMENT (-c) — chars NOT in SET
# ──────────────────────────────────────────
echo "hello123" | tr -cd '0-9' # → 123 (keep ONLY digits)
echo "hello123" | tr -cd 'a-zA-Z' # → hello (keep ONLY letters)
echo "hello123" | tr -cd 'a-zA-Z0-9\n' # Keep alphanumeric + newlines
echo "hello world" | tr -cd '[:alpha:]\n' # Keep letters + newlines
# ──────────────────────────────────────────
# COMBINING FLAGS
# ──────────────────────────────────────────
cat file | tr -cs 'a-zA-Z' '\n' # Words one per line (split on non-alpha)
cat file | tr -cd '[:print:]\n' # Remove non-printable chars
# ──────────────────────────────────────────
# PRACTICAL TR EXAMPLES
# ──────────────────────────────────────────
# Fix Windows CRLF → Unix LF:
tr -d '\r' < windows_file.txt > unix_file.txt
# Convert tabs to spaces:
tr '\t' ' ' < file.txt
# Convert spaces to newlines (one word per line):
echo "one two three" | tr ' ' '\n'
# Convert newlines to spaces (join lines):
tr '\n' ' ' < file.txt
# ROT13 cipher (basic):
echo "hello" | tr 'A-Za-z' 'N-ZA-Mn-za-m' # → uryyb
# Count character frequency:
cat file | tr -cs '[:alpha:]' '\n' | sort | uniq -c | sort -rn
# Generate random password (no special chars):
tr -dc 'A-Za-z0-9' < /dev/urandom | head -c 16; echo
# Generate random password (with special chars):
tr -dc 'A-Za-z0-9!@#$%^&*' < /dev/urandom | head -c 20; echo
# Remove non-ASCII characters:
tr -cd '\11\12\15\40-\176' < file.txt # Keep printable ASCII
6. cut — Extract Fields/Columns
# ==========================================
# CUT — Extract Sections from Lines
# ==========================================
# Cuts out columns or fields from each line.
# Simpler than awk but faster for straightforward column extraction.
# SYNTAX:
# cut [OPTIONS] file
# command | cut [OPTIONS]
# KEY OPTIONS:
# -d DELIM Field delimiter (default: TAB)
# -f FIELDS Select fields (1-based)
# -c CHARS Select by character position
# -b BYTES Select by byte position
# --complement Invert selection
# ──────────────────────────────────────────
# FIELD EXTRACTION (-f with -d)
# ──────────────────────────────────────────
cut -d: -f1 /etc/passwd # Usernames (field 1)
cut -d: -f1,7 /etc/passwd # Username AND shell (fields 1 & 7)
cut -d: -f1-3 /etc/passwd # Fields 1 through 3
cut -d: -f3- /etc/passwd # Field 3 to end
cut -d, -f2 data.csv # Second column of CSV
cut -d'\t' -f3 file.tsv # Third column of TSV
cut -d' ' -f1 /var/log/auth.log # First word of each line
# ──────────────────────────────────────────
# CHARACTER EXTRACTION (-c)
# ──────────────────────────────────────────
cut -c1 file # First character of each line
cut -c1-10 file # Characters 1-10 (first 10 chars)
cut -c5- file # From character 5 to end
cut -c1,5,10 file # Characters 1, 5, and 10
# Extract date components (if format is fixed-width):
echo "2024-03-15" | cut -c1-4 # → 2024 (year)
echo "2024-03-15" | cut -c6-7 # → 03 (month)
echo "2024-03-15" | cut -c9-10 # → 15 (day)
# ──────────────────────────────────────────
# COMPLEMENT (INVERSE SELECTION)
# ──────────────────────────────────────────
cut -d: -f6 --complement /etc/passwd # All fields EXCEPT 6
# ──────────────────────────────────────────
# PRACTICAL CUT EXAMPLES
# ──────────────────────────────────────────
# Extract usernames from /etc/passwd:
cut -d: -f1 /etc/passwd
# Extract UID and username:
cut -d: -f1,3 /etc/passwd
# Get process names from ps:
ps aux | cut -c66- # Command column (character position)
# Extract domain from email addresses:
echo "user@example.com" | cut -d@ -f2 # → example.com
# Get file extensions:
ls *.txt | cut -d. -f2 # → txt
# Extract first word of each line:
cut -d' ' -f1 file.txt
# CSV — print specific columns:
cut -d, -f1,3,5 data.csv # Columns 1, 3, and 5
# CUT vs AWK comparison:
cut -d: -f1 /etc/passwd # Simple — use cut
awk -F: '$3 >= 1000 {print $1}' /etc/passwd # With condition — use awk
7. sort — Sort Lines
# ==========================================
# SORT — Sort Lines of Text
# ==========================================
# Sorts lines alphabetically by default.
# SYNTAX:
# sort [OPTIONS] file
# command | sort [OPTIONS]
# KEY OPTIONS:
# -n Numeric sort (10 > 9, not "10" < "9")
# -r Reverse order (descending)
# -u Unique — remove duplicates (like sort | uniq)
# -k N Sort by field N
# -k N,M Sort from field N to field M
# -t D Field delimiter (default: whitespace)
# -f Case insensitive (fold)
# -h Human-readable numbers (1K, 2M, 3G)
# -V Version sort (v1.9 < v1.10 correctly!)
# -o Write output to file (in-place sort)
# -c Check if sorted (no output, exit code)
# -m Merge already-sorted files
# ──────────────────────────────────────────
# BASIC SORTING
# ──────────────────────────────────────────
sort file.txt # Alphabetical (default)
sort -r file.txt # Reverse alphabetical (Z→A)
sort -n file.txt # Numeric sort
sort -rn file.txt # Reverse numeric (largest first)
sort -f file.txt # Case insensitive
sort -u file.txt # Sort + remove duplicates
sort -h file.txt # Human-readable numbers (1K, 2M)
sort -V file.txt # Version sort (v1.2 < v1.10)
# ──────────────────────────────────────────
# SORT BY FIELD (-k and -t)
# ──────────────────────────────────────────
sort -k2 file # Sort by second field
sort -k3 -n file # Sort by third field numerically
sort -k2 -r file # Sort by second field reverse
sort -t: -k3 -n /etc/passwd # Sort passwd by UID (field 3)
sort -t, -k2 -n data.csv # Sort CSV by second column numerically
sort -k1,1 -k2,2n file # Primary sort on field 1, secondary on field 2 numeric
# ──────────────────────────────────────────
# PRACTICAL SORT EXAMPLES
# ──────────────────────────────────────────
# Sort and deduplicate IPs:
grep -oP '\d+\.\d+\.\d+\.\d+' file | sort -u
# Sort users by UID:
sort -t: -k3 -n /etc/passwd | head -10
# Find largest files:
du -h /var/log/* | sort -rh | head -10
# Sort processes by CPU:
ps aux | sort -k3 -rn | head -10
# Sort processes by memory:
ps aux | sort -k4 -rn | head -10
# Sort CSV by third column:
sort -t, -k3 -rn data.csv
# Count and rank:
cat access.log | awk '{print $1}' | sort | uniq -c | sort -rn | head -20
8. uniq — Remove Duplicates
# ==========================================
# UNIQ — Filter/Count Duplicate Lines
# ==========================================
# Works on ADJACENT duplicates — MUST sort first for full dedup!
# SYNTAX:
# sort file | uniq [OPTIONS]
# command | sort | uniq [OPTIONS]
# KEY OPTIONS:
# -c Prefix count of occurrences
# -d Only print DUPLICATE lines (appeared more than once)
# -u Only print UNIQUE lines (appeared exactly once)
# -i Case insensitive comparison
# -f N Skip first N fields
# -s N Skip first N characters
# ──────────────────────────────────────────
# BASIC USAGE
# ──────────────────────────────────────────
sort file | uniq # Remove duplicates
sort file | uniq -c # Count occurrences
sort file | uniq -d # Show ONLY duplicates
sort file | uniq -u # Show ONLY unique lines
sort -f file | uniq -i # Case insensitive dedup
# ──────────────────────────────────────────
# PRACTICAL UNIQ EXAMPLES
# ──────────────────────────────────────────
# Count and rank most common entries:
awk '{print $1}' access.log | sort | uniq -c | sort -rn | head -10
# Find duplicate lines:
sort file.txt | uniq -d
# Remove duplicate IPs:
grep -oP '\d+\.\d+\.\d+\.\d+' logfile | sort -u
# Count unique users:
cut -d: -f1 /etc/passwd | sort | uniq | wc -l
# Find lines appearing exactly once:
sort file.txt | uniq -u
# Most common HTTP error codes:
awk '{print $9}' access.log | sort | uniq -c | sort -rn
9. wc — Count Words/Lines/Chars
# ==========================================
# WC — Word Count
# ==========================================
# SYNTAX:
# wc [OPTIONS] file
# command | wc [OPTIONS]
# OPTIONS:
# -l Count LINES only
# -w Count WORDS only
# -c Count BYTES only
# -m Count CHARACTERS (multi-byte aware)
# -L Length of longest line
# ──────────────────────────────────────────
# EXAMPLES
# ──────────────────────────────────────────
wc file.txt # Lines, words, bytes, filename
wc -l file.txt # Lines only → "42 file.txt"
wc -l < file.txt # Lines only → "42" (no filename)
wc -w file.txt # Words only
wc -c file.txt # Bytes
wc -m file.txt # Characters
wc -L file.txt # Length of longest line
# Count lines in multiple files:
wc -l *.txt | sort -rn
# Count grep matches:
grep -c 'error' logfile # Built into grep
grep 'error' logfile | wc -l # Same via pipe
# Count running processes:
ps aux | wc -l
# Count files in directory:
ls | wc -l
find . -type f | wc -l
# Count lines of Python code:
find . -name "*.py" -exec cat {} + | wc -l
10. head & tail — First/Last Lines
# ==========================================
# HEAD — Print First Lines
# TAIL — Print Last Lines
# ==========================================
# HEAD:
head file.txt # First 10 lines (default)
head -n 20 file.txt # First 20 lines
head -n -5 file.txt # All EXCEPT last 5 lines
head -c 100 file.txt # First 100 BYTES
head -c 1K file.txt # First 1 kilobyte
# TAIL:
tail file.txt # Last 10 lines (default)
tail -n 20 file.txt # Last 20 lines
tail -n +5 file.txt # All lines FROM line 5 onwards
tail -c 100 file.txt # Last 100 bytes
tail -f file.txt # FOLLOW — real-time updates (log monitoring!)
tail -F file.txt # Follow + retry if file recreated
# PRACTICAL EXAMPLES:
head -1 file.txt # Print header line only
tail -1 file.txt # Print last line only
tail -f /var/log/syslog # Live log monitoring
tail -f /var/log/auth.log | grep 'Failed' # Monitor failed logins live!
# Skip header of CSV:
tail -n +2 data.csv # Skip first line (header), process rest
# Print lines 5-10:
sed -n '5,10p' file # sed approach
awk 'NR>=5 && NR<=10' file # awk approach
head -10 file | tail -6 # head/tail combo
11. paste — Merge Lines/Files
# ==========================================
# PASTE — Merge Lines Side by Side
# ==========================================
# SYNTAX:
# paste [OPTIONS] file1 file2 ...
# command | paste [OPTIONS]
# OPTIONS:
# -d DELIM Delimiter between merged fields (default: TAB)
# -s Serial — paste one file at a time (transpose)
# EXAMPLES:
# Merge two files side by side:
paste file1.txt file2.txt # Tab-separated columns
paste -d, file1.txt file2.txt # Comma-separated
paste -d'|' file1.txt file2.txt # Pipe-separated
# Join lines of a single file:
paste - - < file.txt # Two lines per output line
paste -d, - - - < file.txt # Three lines as CSV
# Transpose rows (serial mode):
paste -s file.txt # One file, columns joined to one row
paste -sd, file.txt # Comma-join all lines → "a,b,c,d"
# Build a CSV from two columns:
cut -d: -f1 /etc/passwd > users.txt
cut -d: -f3 /etc/passwd > uids.txt
paste -d, users.txt uids.txt # → "root,0\ndaemon,1..."
# Practical — convert list to comma-separated:
cat list.txt | paste -sd, - # → "item1,item2,item3"
12. tee — Split Output
# ==========================================
# TEE — Read stdin, Write to stdout AND file
# ==========================================
# Like a T-junction in a pipe — output goes two directions.
# SYNTAX:
# command | tee [OPTIONS] file(s)
# OPTIONS:
# -a Append to file (don't overwrite)
# -i Ignore interrupt signals
# EXAMPLES:
# Save output AND see it on screen:
ls -la | tee output.txt # View + save simultaneously
grep 'error' log | tee errors.txt # Filter + save
# Write to multiple files:
cat file | tee copy1.txt copy2.txt # Save to two files at once
# Append mode:
echo "new line" | tee -a file.txt # Append to file
# In a pipeline (save intermediate result):
cat access.log | grep '404' | tee 404s.txt | awk '{print $1}' | sort | uniq -c
# Use tee with sudo (write to root-owned file):
echo "text" | sudo tee /etc/file # Avoids "sudo echo > file" issue
echo "PermitRootLogin no" | sudo tee -a /etc/ssh/sshd_config
13. xargs — Build & Execute Commands
# ==========================================
# XARGS — Build and Execute Command Lines
# ==========================================
# Takes stdin input and passes it as ARGUMENTS to another command.
# Bridges the gap between commands that output text and commands
# that take arguments (not stdin).
# SYNTAX:
# command | xargs [OPTIONS] [COMMAND]
# KEY OPTIONS:
# -n N Use at most N arguments per command invocation
# -I {} Replace {} with each input item (one at a time)
# -P N Run N processes in parallel
# -0 Input items are NUL-separated (for filenames with spaces)
# -d DELIM Use DELIM as input delimiter
# -t Print command before executing (verbose)
# -p Prompt before executing each command
# -r Don't run if stdin is empty
# ──────────────────────────────────────────
# BASIC EXAMPLES
# ──────────────────────────────────────────
# Remove all .tmp files:
find . -name "*.tmp" | xargs rm
# Print each file's line count:
find . -name "*.txt" | xargs wc -l
# grep across found files:
find . -name "*.py" | xargs grep -l 'import os'
# ──────────────────────────────────────────
# -I {} — REPLACE MODE (most flexible)
# ──────────────────────────────────────────
# Copy each file to backup:
ls *.conf | xargs -I{} cp {} {}.bak
# Move files to directory:
cat filelist.txt | xargs -I{} mv {} /backup/
# Run command for each line:
cat servers.txt | xargs -I{} ssh {} 'uptime'
# Create directories from list:
cat dirs.txt | xargs -I{} mkdir -p {}
# ──────────────────────────────────────────
# -n N — BATCH MODE
# ──────────────────────────────────────────
echo "a b c d e f" | xargs -n2 echo # 2 args per call:
# → echo a b
# → echo c d
# → echo e f
cat packages.txt | xargs -n1 apt-get install -y # Install one by one
# ──────────────────────────────────────────
# -P N — PARALLEL EXECUTION
# ──────────────────────────────────────────
# Run 4 parallel ping tests:
cat hosts.txt | xargs -P4 -I{} ping -c1 {}
# Parallel file processing:
find . -name "*.log" | xargs -P8 -I{} gzip {}
# ──────────────────────────────────────────
# -0 — NULL DELIMITER (safe for filenames)
# ──────────────────────────────────────────
# Safe handling of filenames with spaces:
find . -name "*.txt" -print0 | xargs -0 grep 'pattern'
find . -type f -print0 | xargs -0 chmod 644
# ──────────────────────────────────────────
# PRACTICAL XARGS EXAMPLES
# ──────────────────────────────────────────
# Mass rename files:
ls *.txt | xargs -I{} mv {} {%.txt}.md
# Check if hosts are up:
cat hosts.txt | xargs -P10 -I{} sh -c 'ping -c1 -W1 {} >/dev/null 2>&1 && echo "{} UP" || echo "{} DOWN"'
# Download multiple URLs:
cat urls.txt | xargs -P5 -I{} wget -q {}
# Search pattern in all git-tracked files:
git ls-files | xargs grep -l 'TODO'
# Delete files older than 30 days:
find /tmp -mtime +30 -type f | xargs rm -f
14. Combining Tools in Pipelines
# ==========================================
# THE UNIX PIPELINE — WHERE REAL POWER LIVES
# ==========================================
# Each tool does ONE thing well.
# Combine them with | (pipe) for complex processing.
# Pipeline anatomy:
# SOURCE | FILTER | TRANSFORM | AGGREGATE | OUTPUT
# cat grep awk sort head
# ──────────────────────────────────────────
# CLASSIC COMBINATIONS
# ──────────────────────────────────────────
# Top 10 IPs in access log:
cat access.log | awk '{print $1}' | sort | uniq -c | sort -rn | head -10
# Find most common words in a file:
cat file.txt | tr '[:upper:]' '[:lower:]' | tr -cs '[:alpha:]' '\n' | sort | uniq -c | sort -rn | head -20
# Count lines matching pattern:
grep -c 'ERROR' logfile
grep 'ERROR' logfile | wc -l # Same
# All users with bash shell sorted:
grep '/bash$' /etc/passwd | cut -d: -f1 | sort
# Extract and deduplicate emails:
grep -oE '[a-zA-Z0-9._%+\-]+@[a-zA-Z0-9.\-]+\.[a-zA-Z]{2,}' file | sort -u
# Process CSV — sum third column:
cut -d, -f3 data.csv | tail -n +2 | awk '{sum+=$1} END{print sum}'
# Find files, grep them, show results:
find /var/www -name "*.php" | xargs grep -lP 'eval\(base64_decode'
# ──────────────────────────────────────────
# ADVANCED PIPELINE PATTERNS
# ──────────────────────────────────────────
# Real-time log error monitor:
tail -f /var/log/syslog | grep --line-buffered 'ERROR' | awk '{print strftime("%H:%M:%S"), $0}'
# Build report from log:
awk '{print $9}' access.log | sort | uniq -c | sort -rn | \
awk 'BEGIN{print "Status\tCount"} {print $2"\t"$1}' | \
column -t
# Find and replace in multiple files:
grep -rl 'oldstring' . | xargs sed -i 's/oldstring/newstring/g'
# Extract all links from HTML file:
grep -oP 'href="[^"]*"' page.html | sed 's/href="//;s/"//' | sort -u
# Count code lines by language:
find . -type f | sed 's/.*\.//' | sort | uniq -c | sort -rn
# Disk usage by type in /var/log:
find /var/log -type f | sed 's/.*\.//' | sort | \
xargs -I{} sh -c 'find /var/log -name "*.{}" -exec du -sb {} + 2>/dev/null | awk "{s+=\$1} END{print s, \"{}\"}"'
# ──────────────────────────────────────────
# STREAM EDITING IN PIPELINE
# ──────────────────────────────────────────
# Transform and filter in one pipeline:
cat /etc/passwd | \
grep -v '^#' | \ # Remove comments
cut -d: -f1,3,7 | \ # Username, UID, shell
grep '/bash$' | \ # Only bash users
sort -t: -k2 -n | \ # Sort by UID
awk -F: '{printf "%-15s UID=%-6s Shell=%s\n",$1,$2,$3}' # Format
# Log analysis pipeline:
cat access.log | \
awk '$9 >= 400' | \ # Only error responses
awk '{print $1,$9,$7}' | \ # IP, status, URL
sort | uniq -c | sort -rn | \# Count + rank
head -20 # Top 20
15. Real-World Use Cases
# ==========================================
# REAL-WORLD SCENARIOS
# ==========================================
# ──────────────────────────────────────────
# LOG ANALYSIS
# ──────────────────────────────────────────
# Top attacking IPs:
grep 'Failed password' /var/log/auth.log | \
grep -oP 'from \K[\d.]+' | sort | uniq -c | sort -rn | head -20
# Successful logins:
grep 'Accepted' /var/log/auth.log | awk '{print $9, $11}' | sort -u
# Apache error analysis:
awk '$9 >= 400 {print $9, $7}' /var/log/apache2/access.log | \
sort | uniq -c | sort -rn | head -20
# Nginx — bandwidth per IP:
awk '{bytes[$1]+=$10} END{for(ip in bytes) print bytes[ip], ip}' access.log | \
sort -rn | head -10
# ──────────────────────────────────────────
# SYSTEM ADMINISTRATION
# ──────────────────────────────────────────
# Find large log files:
find /var/log -type f -exec du -h {} + | sort -rh | head -10
# Users with no password:
awk -F: '$2=="" {print $1}' /etc/shadow
# List all open ports and their processes:
ss -tlnp | awk 'NR>1 {print $4, $6}' | column -t
# Disk usage by directory:
du -sh /* 2>/dev/null | sort -rh | head -15
# Find files modified in last 24h:
find / -mtime -1 -type f 2>/dev/null | head -20
# Monitor specific process memory:
while true; do
ps aux | grep nginx | grep -v grep | awk '{print $4"%", $6"KB", $11}'
sleep 5
done
# ──────────────────────────────────────────
# PENTESTING / CTF
# ──────────────────────────────────────────
# Find potential credentials in files:
grep -rP '(password|passwd|pwd|secret|api_key)\s*[=:]\s*\S+' . 2>/dev/null
# Extract all IPs from output:
command_output | grep -oP '\b\d{1,3}(\.\d{1,3}){3}\b' | sort -u
# Extract and decode base64:
grep -oP '[A-Za-z0-9+/]{20,}={0,2}' file | while read b64; do
echo "$b64" | base64 -d 2>/dev/null
echo "---"
done
# Find SUID binaries and cross-reference GTFOBins:
find / -perm -4000 -type f 2>/dev/null | xargs ls -la
# Extract URLs from web response:
curl -s http://target.com | grep -oP 'https?://[^\s"'"'"'<>]+' | sort -u
# Generate wordlist from webpage:
curl -s http://target.com | grep -oE '[a-zA-Z]{4,}' | tr '[:upper:]' '[:lower:]' | sort -u > wordlist.txt
# Parse nmap output:
cat scan.gnmap | grep 'open' | awk '{print $2, $5}' | \
sed 's|/.*||' | column -t | sort -k2 -n
# ──────────────────────────────────────────
# DATA PROCESSING
# ──────────────────────────────────────────
# Convert CSV to TSV:
sed 's/,/\t/g' data.csv > data.tsv
tr ',' '\t' < data.csv > data.tsv # With tr
# Remove CSV header and sum column 3:
tail -n +2 data.csv | cut -d, -f3 | awk '{sum+=$1} END{print sum}'
# Count unique values in CSV column 2:
cut -d, -f2 data.csv | tail -n +2 | sort | uniq -c | sort -rn
# Find and fix encoding issues:
file *.txt | grep -v 'UTF-8\|ASCII' | cut -d: -f1 | \
xargs -I{} iconv -f latin1 -t utf-8 {} -o {}.fixed
16. Master Cheat Sheet
╔═══════════════════════════════════════════════════════════════════╗
║ LINUX TEXT PARSING TOOLS — QUICK REFERENCE ║
╚═══════════════════════════════════════════════════════════════════╝
TOOL SELECTION GUIDE:
─────────────────────────────────────────────────────────────────
Search/filter lines? → grep 'pattern' file
Case insensitive search? → grep -i 'pattern' file
Invert match? → grep -v 'pattern' file
Count matches? → grep -c 'pattern' file
Show line numbers? → grep -n 'pattern' file
Only matched text? → grep -o 'pattern' file
Recursive search? → grep -r 'pattern' dir/
PCRE regex (lookahead etc.)? → grep -P 'pattern' file
Find + replace text? → sed 's/old/new/g' file
Delete lines? → sed '/pattern/d' file
Print specific lines? → sed -n '5,10p' file
In-place file editing? → sed -i 's/old/new/g' file
Extract fields/columns? → awk '{print $2}' file
CSV/TSV processing? → awk -F, '{print $3}' file
Math on columns? → awk '{sum+=$3} END{print sum}' file
Simple column extract? → cut -d: -f1 file
Change character cases? → tr '[:lower:]' '[:upper:]'
Delete characters? → tr -d '0-9'
Remove duplicates? → sort | uniq
Count occurrences? → sort | uniq -c | sort -rn
Sort numerically? → sort -n
Sort largest first? → sort -rn
Count lines? → wc -l
First N lines? → head -n N
Last N lines? → tail -n N
Follow live file? → tail -f
Run cmd on each input? → xargs -I{} cmd {}
Save + see output? → tee file.txt
GREP FLAGS:
-i insensitive -v invert -n line numbers
-c count -l filenames -o match only
-r recursive -w word -A/-B/-C context
-E ERE -P PCRE -q quiet (exit code)
SED COMMANDS:
s/old/new/g Substitute (g=all, i=nocase, 2=2nd)
/pat/d Delete matching lines
/pat/p Print matching lines (with -n)
/pat/i\TEXT Insert before match
/pat/a\TEXT Append after match
= Print line number
y/abc/ABC/ Transliterate (like tr)
\1 \2 Backreferences in replacement
& Entire matched text
AWK VARIABLES:
$0 entire line $1..$NF fields NR line number
NF field count FS input sep OFS output sep
FILENAME current file
AWK FUNCTIONS:
print/printf length() toupper()/tolower()
substr() split() sub()/gsub()
match() index() sprintf()
COMMON PATTERNS:
IP address: [0-9]{1,3}(.[0-9]{1,3}){3}
Email: [a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+.[a-zA-Z]{2,}
URL: https?://[^\s"'<>]+
Date: [0-9]{4}-[0-9]{2}-[0-9]{2}
MD5: [a-fA-F0-9]{32}
Blank line: ^$
Comment line: ^[[:space:]]*#
PIPELINE TEMPLATES:
Count + rank: | sort | uniq -c | sort -rn
Top 10: | sort | uniq -c | sort -rn | head -10
Unique sort: | sort -u
Extract IPs: grep -oP '\d+.\d+.\d+.\d+'
Field + rank: awk '{print $N}' | sort | uniq -c | sort -rn
Mass replace: grep -rl 'old' . | xargs sed -i 's/old/new/g'
Mass delete: find . -name "*.tmp" | xargs rm -f