Remove duplicate lines from file using awk
We mostly have the requirement to remove duplicate lines from file, support guys know the pain well
Lets look at the quickest solution to achieve this.
awk ‘!x[$0]++’ file1.txt
x[$0]: look at the value of key
$0, in associative array x. If it does not exist, create it.
x[$0]++: increment the value of x
[$0], return the old value as value of expression. If x
[$0]does not exist, return
0and increment x
++operator returns numeric value).
!x[$0]++: negate the value of expression. If x
0, the whole expression is evaluated to true, make
awkperformed default action
print $0. Otherwise, the whole expression is evaluated to false, causes