100 Useful Command-Line Utilities

by Oliver; 2014

42. sed

From An Introduction to the Command-Line (on Unix-like systems) - sed: Sed, like awk, is a full-fledged language that is convenient to use in a very limited sphere (GNU Sed Guide). I mainly use it for two things: (1) replacing text, and (2) deleting lines. Sed is often mentioned in the same breath as regular expressions although, like the rest of the world, I'd use Perl and Python when it comes to that. Nevertheless, let's see what sed can do.

Sometimes the first line of a text file is a header and you want to remove it. Then:

$ cat test_header.txt
This is a header
1	asdf
2	asdf
2	asdf

$ cat test_header.txt | sed '1d'    # delete the first line
1	asdf
2	asdf
2	asdf

To remove the first 3 lines:

$ cat test_header.txt | sed '1,3d'  # delete lines 1-3
2	asdf

1,3 is sed's notation for the range 1 to 3. We can't do much more without entering regular expression territory. One sed construction is: where d stands for delete if the pattern is matched. So to remove lines beginning with #:

$ cat test_comment.txt 
1	asdf
# This is a comment
2	asdf
# This is a comment
2	asdf

$ cat test_comment.txt | sed '/^#/d'
1	asdf
2	asdf
2	asdf

Another construction is: where s stands for substitute. So this means replace A with B. By default, this only works for the first occurrence of A, but if you put a g at the end, for group, all As are replaced: For example:

$ # replace 1st occurrence of kitty with X
$ echo "hello kitty. goodbye kitty" | sed 's/kitty/X/'  
hello X. goodbye kitty

$ # same thing. using | as a separator is ok
$ echo "hello kitty. goodbye kitty" | sed 's|kitty|X|'  
hello X. goodbye kitty

$ # replace all occurrences of kitty
$ echo "hello kitty. goodbye kitty" | sed 's/kitty/X/g' 
hello X. goodbye X

This is such a useful ability that all text editors allow you to perform find-and-replace as well. By replacing some text with nothing, you can also use this as a delete:

$ echo "hello kitty. goodbye kitty" | sed 's|kitty||'
hello . goodbye kitty

$ echo "hello kitty. goodbye kitty" | sed 's|kitty||g'
hello . goodbye 

Sed is especially good when you're trying to rename batches of files on the command line. I often have occasion to use it in for loops:

$ touch file1.txt file2.txt file3.txt
$ for i in file*; do echo $i; j=$( echo $i | sed 's|.txt|.html|'); mv $i $j; done
file1.txt
file2.txt
file3.txt
$ ls
file1.html  file2.html	file3.html

Sed has the ability to edit files in place with the -i flag—that is to say, modify the file wihout going through the trouble of creating a new file and doing the re-naming dance. For example, to add the line This is a header to the top of myfile.txt:

$ sed -i '1i This is a header' myfile.txt

Delete the first line—e.g., a header—from a file:

$ sed -i '1d' myfile.txt

<PREV   NEXT>