Remove duplicate lines from file using awk
We mostly have the requirement to remove duplicate lines from file, support guys know the pain well 🙂
Lets look at the quickest solution to achieve this.
awk ‘!x[$0]++’ file1.txt
x[$0]: look at the value of key
$0, in associative array x. If it does not exist, create it.
x[$0]++: increment the value of x
[$0], return the old value as value of expression. If x
[$0]does not exist, return
0and increment x
++operator returns numeric value).
!x[$0]++: negate the value of expression. If x
0, the whole expression is evaluated to true, make
awkperformed default action
print $0. Otherwise, the whole expression is evaluated to false, causes