by Oliver; 2014
46. uniq
uniq filters a file leaving only the unique lines, provided the file is sorted.
Suppose:
$ cat test.txt
aaaa
bbbb
aaaa
aaaa
cccc
cccc
Then:
$ cat test.txt | uniq
aaaa
bbbb
aaaa
cccc
This can be thought of as a local uniquing—adjacent rows are not the same, but you can still have a repeated row in the file.
If you want the global unique, sort first:
$ cat test.txt | sort | uniq
aaaa
bbbb
cccc
This is identical to:
$ cat test.txt | sort -u
aaaa
bbbb
cccc
uniq also has the ability to show you only the lines that are
not unique with the
duplicate flag:
$ cat test.txt | sort | uniq -d
aaaa
cccc
And
uniq can count the number of distinct rows in a file (provided it's sorted):
cat test.txt | sort | uniq -c
3 aaaa
1 bbbb
2 cccc
<PREV
NEXT>
