干货来源于酷壳的**AWK 简明教程以及Linux与unix shell编程指南**。

本文纯属搬砖。

最近经常遇到需要处理大段文本输出的情况，所以想学习一下awk命令，这个命令更像是一个微型语言。

《awk命令简介（一）》先着重介绍awk的应用场景，内容主要来自酷壳的**AWK 简明教程**。

awk快速上手#

待处理的文本#

1
$ netstat -a > output
2
$ cat output | more -n 6
3
$ Proto Recv-Q Send-Q  Local Address          Foreign Address        (state)
4
tcp4       0      0  192.168.1.100.62750    114.112.202.20.http    SYN_SENT
5
tcp4       0      0  192.168.1.100.62749    113.103.139.251.10684  CLOSE_WAIT
6
tcp4       0      0  192.168.1.100.62748    202.150.16.5.cslistene ESTABLISHED
7
tcp4       0      0  192.168.1.100.62747    114.112.202.20.http    ESTABLISHED
8
tcp4       0      0  192.168.1.100.62746    114.82.234.97.12077    SYN_SENT

最简单的选择列，输出第一列`$1`与第四列`$4`#

1
$ awk '{print $1, $4}' output
2
Proto Local-Address
3
tcp4 192.168.1.100.62750
4
tcp4 192.168.1.100.62749
5
tcp4 192.168.1.100.62748
6
tcp4 192.168.1.100.62747

C风格的格式化输出#

1
$ awk '{printf "%-8s %-8s %-8s %-18s %-22s %-15s\n", $1, $2, $3, $4, $5, $6}' output
2
Proto    Recv-Q   Send-Q   Local-Address      Foreign-Address        (state)
3
tcp4     0        0        192.168.1.100.62750 114.112.202.20.http    SYN_SENT
4
tcp4     0        0        192.168.1.100.62749 113.103.139.251.10684  CLOSE_WAIT
5
tcp4     0        0        192.168.1.100.62748 202.150.16.5.cslistene ESTABLISHED
6
tcp4     0        0        192.168.1.100.62747 114.112.202.20.http    ESTABLISHED

过滤条目#

取第三列为0且第6列位“LISTEN”的数据：

1
$ awk '$3==0 && $6=="LISTEN"' output
2
tcp4       0      0  *.oms                  *.*                    LISTEN
3
tcp4       0      0  *.socks                *.*                    LISTEN
4
tcp4       0      0  *.58932                *.*                    LISTEN
5
tcp4       0      0  localhost.ddi-tcp-1    *.*                    LISTEN
6
tcp4       0      0  localhost.51219        *.*                    LISTEN

同时用内建变量NR选出表头（第一行）：

1
$ awk '$3==0 && $6=="LISTEN" || NR==1' output
2
Proto Recv-Q Send-Q  Local-Address          Foreign-Address        (state)
3
tcp4       0      0  *.oms                  *.*                    LISTEN
4
tcp4       0      0  *.socks                *.*                    LISTEN
5
tcp4       0      0  *.58932                *.*                    LISTEN
6
tcp4       0      0  localhost.ddi-tcp-1    *.*                    LISTEN

取第三列大于0的数据：

1
$ awk ' $3>0 {print $0}' output
2
Proto Recv-Q Send-Q  Local-Address          Foreign-Address        (state)
3
tcp4       0     68  192.168.1.100.62743    123.138.206.178.8948   ESTABLISHED
4
tcp4       0     68  192.168.1.100.62720    123.138.54.160.mdqs    ESTABLISHED
5
tcp4       0      9  192.168.1.100.62707    180.97.152.55.http-alt FIN_WAIT_1
6
tcp4       0     68  192.168.1.100.62695    123.138.206.178.8948   FIN_WAIT_1

取第三列为0且第六列为“LISTEN”的数据，并进行格式输出：

1
$ awk '$3==0 && $6=="LISTEN" || NR==1 {printf "%-20s %-20s %s\n",$4,$5,$6}' output
2
Local-Address        Foreign-Address      (state)
3
*.oms                *.*                  LISTEN
4
*.socks              *.*                  LISTEN
5
*.58932              *.*                  LISTEN
6
localhost.ddi-tcp-1
7
*.*                  LISTEN

内置变量名称	功能描述
$0	当前记录（行）
$1~$ n	当前行的第n个字段（列）
FS	输入字段（列）分隔符，默认为space或tab
NF	当前记录（行）的字段（列）数
NR	总记录（行）数，多文件记录（行）数累加，不清零
FNR	当前文件记录（行）数，每文件清零
RS	输入换行符，默认系统换行符
OFS	输出字段（列）分隔符，默认为space
ORS	输出换行符，默认系统换行符
FILENAME	当前文件名

输出行号

1
$ awk '$3==0 && $6=="ESTABLISHED" || NR==1 {printf "%02s %s %-20s %-20s %s %s", NR, FNR, $4, $5, $6, ORS}' output
2
01 1 Local-Address        Foreign-Address      (state)
3
04 4 192.168.1.100.62748  202.150.16.5.cslistene ESTABLISHED
4
05 5 192.168.1.100.62747  114.112.202.20.http  ESTABLISHED
5
07 7 192.168.1.100.62744  114.112.202.20.http  ESTABLISHED
6
11 11 192.168.1.100.62739  14.157.50.39.13566   ESTABLISHED

指定输入字段（列）分隔符

1
$ awk  'BEGIN{FS=":"} {print $1,$3,$6}' /etc/passwd
2
nobody -2 /var/empty
3
root 0 /var/root
4
daemon 1 /var/root
5
_uucp 4 /var/spool/uucp
6
_taskgated 13 /var/empty

也可以用-F指定

1
$ awk  -F: '{print $1,$3,$6}' /etc/passwd

或指定多种分隔符

1
$ awk  -F '[;:]' '{print $1,$3,$6}' /etc/passwd

指定输出字段（列）分隔符

1
$ awk  -F: '{print $1,$3,$6}' OFS="\t" /etc/passwd
2
nobody  -2      /var/empty
3
root    0       /var/root
4
daemon  1       /var/root
5
_uucp   4       /var/spool/uucp
6
_taskgated      13      /var/empty

字符串匹配#

~表示匹配开始，//中是模式，类似正则表达式

匹配第六个字段（列）有“FIN”的记录（行）

1
$ awk '$6 ~ /FIN/ || NR==1 {printf "%-4s %-20s %-30s %-10s\n", NR, $4, $5, $6}' output
2
1    Local-Address        Foreign-Address                (state)
3
21   192.168.1.100.62707  180.97.152.55.http-alt         FIN_WAIT_1
4
22   192.168.1.100.62695  123.138.206.178.8948           FIN_WAIT_1
5
24   192.168.1.100.62690  218.86.133.247.8368            FIN_WAIT_1
6
25   192.168.1.100.62689  14.157.50.39.13566             FIN_WAIT_1

或是有“WAIT”的记录

1
$ awk '$6 ~ /WAIT/ || NR==1 {printf "%-4s %-20s %-30s %-10s\n", NR, $4, $5, $6}' output
2
1    Local-Address        Foreign-Address                (state)
3
3    192.168.1.100.62749  113.103.139.251.10684          CLOSE_WAIT
4
21   192.168.1.100.62707  180.97.152.55.http-alt         FIN_WAIT_1
5
22   192.168.1.100.62695  123.138.206.178.8948           FIN_WAIT_1
6
24   192.168.1.100.62690  218.86.133.247.8368            FIN_WAIT_1

或是直接匹配一整条记录（行）

1
$ awk '/LISTEN/' output
2
tcp4       0      0  *.oms                  *.*                    LISTEN
3
tcp4       0      0  *.socks                *.*                    LISTEN
4
tcp4       0      0  *.58932                *.*                    LISTEN
5
tcp4       0      0  localhost.ddi-tcp-1    *.*                    LISTEN
6
tcp4       0      0  localhost.51219        *.*                    LISTEN

像正则表达式一样，使用|表示“或”

1
$ awk '$6 ~ /CLOSE|LAST/ {printf "%-4s %-20s %-30s %-10s\n", NR, $4, $5, $6}' output
2
3    192.168.1.100.62749  113.103.139.251.10684          CLOSE_WAIT
3
10   192.168.1.100.62741  114.82.234.97.12077            LAST_ACK
4
13   192.168.1.100.62729  114.82.234.97.12077            LAST_ACK
5
16   192.168.1.100.62722  114.82.234.97.12077            LAST_ACK
6
19   192.168.1.100.62711  113.103.139.251.10684          LAST_ACK

用!取反

1
$ awk '$6 !~ /WAIT/ {printf "%-4s %-20s %-30s %-10s\n", NR, $4, $5, $6}' output
2
1    Local-Address        Foreign-Address                (state)
3
2    192.168.1.100.62750  114.112.202.20.http            SYN_SENT
4
4    192.168.1.100.62748  202.150.16.5.cslistene         ESTABLISHED
5
5    192.168.1.100.62747  114.112.202.20.http            ESTABLISHED
6
6    192.168.1.100.62746  114.82.234.97.12077            SYN_SENT

也可以

1
$ awk '!/WAIT/' output
2
Proto Recv-Q Send-Q  Local-Address          Foreign-Address        (state)
3
tcp4       0      0  192.168.1.100.62750    114.112.202.20.http    SYN_SENT
4
tcp4       0      0  192.168.1.100.62748    202.150.16.5.cslistene ESTABLISHED
5
tcp4       0      0  192.168.1.100.62747    114.112.202.20.http    ESTABLISHED
6
tcp4       0      0  192.168.1.100.62746    114.82.234.97.12077    SYN_SENT

拆分文件#

按照第六个字段（列）拆分文件

1
$ awk 'NR!=1 {print > $6}' output
2
$ ls
3
CLOSE_WAIT  ESTABLISHED FIN_WAIT_1  FIN_WAIT_2  LAST_ACK    LISTEN      SYN_SENT    output

也可以指定要输出的字段（列）
Terminal window
```
1
$ awk 'NR!=1 {print $4, $5 > $6}' output
```

或是更加复杂，注意quote>说明了awk是一个脚本解释器

1
$ awk 'NR!=1{if($6 ~ /TIME|ESTABLISHED/) print > "1.txt";
2
quote> else if($6 ~ /LISTEN/) print > "2.txt";
3
quote> else print > "3.txt" }' output
4

5
$ ls ?.txt
6
1.txt  2.txt  3.txt

统计#

统计当前目录下.py, .pyc, .pyo文件的总大小

1
$ ls -l *.py *.pyc *.pyo | awk '{sum+=$5} END {print sum}'
2
11609730

或是按照第六个字段（列）统计，类似GROUP BY，其中a类似Python中的dict（键-值）

1
$ awk 'NR!=1 {a[$6]++;} END {for (i in a) print i ", " a[i];}' output
2
LISTEN, 12
3
FIN_WAIT_1, 51
4
FIN_WAIT_2, 23
5
SYN_SENT, 3常见的正则表达式功能
6
LAST_ACK, 21
7
CLOSE_WAIT, 5
8
ESTABLISHED, 20

统计每个用户的进程所占用的内存信息

1
$ aux | awk 'NR!=1 {a[$1]+=$6;} END { for(i in a) print i ", " a[i]"KB";}'
2
root, 718624KB
3
_usbmuxd, 2768KB
4
_locationd, 7780KB
5
_mdnsresponder, 3760KB
6
zealot, 3800464KB

-EOF-