Lab No. 7: Regular Expressions¶

Match any character¶

In regular expressions, the dot character (.) is used to match any character (with the exception of the new line character)

The expression .a will match any line that has an a character in the middle. For example, it will match “It’s Thursday”. Note how in this example the match includes both the d and the a. In this case the . is matched by the d character.

Note that in our example the expression calls for a character, any character, before the a. This means that it will not match a string such as “abcdefg”.

Anchors¶

Anchors are used to specify if the regular expression needs to match the beginning of the line by using the caret character (^) or the end of the line by using the dollar sign character ($).

As an example, the expression ^Th matches the sentence “This is it” but it does not match “Today is Thursday”

Bracket Expressions and Negation¶

Brackets allow matching a character from a predetermined set of characters. This functionality is almost identical to the pathname expansion list of characters expression. Take for example the expression h[eo]ard.

[user@blue ~]$ echo "I heard." | grep 'h[eo]ard'
I heard.
[user@blue ~]$ echo "I hoard." | grep "h[eo]ard"
I hoard.

When you want to the opposite type of matching, where you want to match any character that is not part of a set of characters, you need to use the caret character (^) as the first character within the brackets. Note in the following example how only pet, put and pit were matched (highlighted).

[user@blue ~]$ echo "I pat my pet and then I put the pot in the pit" | grep 'p[^ao]t'
I pat my pet and then I put the pot in the pit

We can see that the input line matches, and that we have three different matches. If you run the same command and enable the -o option which prints only the matching regions and prints every region on its own line, you’ll be able to see the individual matches:

[user@blue ~]$ echo "I pat my pet and then I put the pot in the pit" | grep -o 'p[^ao]t'
pet
put
pit

Just in the same fashion as the pathname expansion counterpart, you can specify ranges of characters. For example, to specify letters ranging from a to k, you can apply the expression [d-k]:

[user@blue ~]$ echo "abcdefghijklmnopqrstuvwxyz" | grep '[d-k]'
abcdefghijklmnopqrstuvwxyz

It is clear from the example that the string “abcdefghijklmnopqrstuvwxyz” matches the expression [d-k]. Something that is not obvious is that the string matches the pattern 8 different times, one for each of the characters included in the set [d-k]. So the match is not “defghijk” but instead the individual characters “d”, “e”, … , “k”. In this context where we are looking at the whole sentence so this might not seem to make a big difference, but when you apply regular expressions and need to extract matched regions the difference becomes evident.

[user@blue ~]$ echo "abcdefghijklmnopqrstuvwxyz" | grep -o '[d-k]'
d
e
f
g
h
i
j
k

What if you need to match a character to a set that includes several ranges that are non contiguous? In that case you can simple add them in the brackets, without a space between them. A recurring need is to match any alpha-numerical character, lowercase and uppercase. The expression [A-Za-z0-9] will serve this purpopse.

Matching numbers is a little trickier. The main problem is that regular expressions deal with text instead of numeric values, which means that if we want to match a number in a numeric range is not as simple as specifing the beginning and the end of the range. For example, you can not use the range [0-20] to specify a range of numbers between 0 and 20. We’ll cover this later once we have seen alternation

Character Classes¶

Regular expressions developers realized that there are a few patterns that are commonly used over and over. So they came with the idea of defining some predetermined lists of characters. The following table summarizes the most common classes (shamefully adapted from out Textbook The Linux Command Line by William Shotts)

Alternation¶

Aternation is a feature that allows applying alternative patterns at the same time. For example the expression R2-D2|C3PO:

[user@blue ~]$ echo 'The droid you are looking for is R2-D2' | grep -E 'R2-D2|C3PO'
The droid you are looking for is R2-D2
[user@blue ~]$ echo 'The droid you are looking for is C3PO' | grep -E 'R2-D2|C3PO'
The droid you are looking for is C3PO

Alternation is an extended feature of the standard set of regular expression features of the grep command. Because of this, we had to use the -E option in the previous example.

[user@blue ~]$ cat /bin/egrep
#!/bin/sh
exec grep -E "$@"

Now that we know how to use alternation, we can apply it to create expressions that match numeric ranges. Consider a list of numbers, each one on it’s own line:

[user@blue ~]$ printf '%s\n' {0..15}
0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

Suppose we want to match numbers between 7 and 13. The first set of numbers is from 7 to 9, so we can use the expression [7-9] for that purpose. The next set of numbers, from 10 to 13 can be matched by the expression 1[0-3]. We can use alternation to match the desired range:

[user@blue ~]$ printf '%s\n' {0..15} | grep -E '^[7-9]$|^1[0-3]$'
7
8
9
10
11
12
13

Quantifiers¶

Quantifiers are also an extended regular expression feature. The following table list quantifiers and their purpose:

Inverting the match¶

You can use the -v option of grep to instead of printing the lines that match, to print the lines that do not match:

[user@blue ~]$ printf '%s\n' {0..15} | grep -vE '^[7-9]$|^1[0-3]$'
0
1
2
3
4
5
6
14
15

Matched lines count¶

Whenever you are interested in counting the lines that match a pattern, you can use the -c option, which saves you a call to the wc command. For example this:

[user@blue ~]$ printf '%s\n' {0..15} | grep -E '^[7-9]$|^1[0-3]$' | wc -l
7

Can be substituted by this:

[user@blue ~]$ printf '%s\n' {0..15} | grep -cE '^[7-9]$|^1[0-3]$'
7

Note however that the -c is really applied to counting lines, not the individual match counts:

[user@blue ~]$ echo "abcdefghijklmnopqrstuvwxyz" | grep -oc '[d-k]'
1
[user@blue ~]$ echo "abcdefghijklmnopqrstuvwxyz" | grep -c '[d-k]'
1

Grep with multiple files¶

When grep has a list of files specified as parameters, it prints every match on a line preceded by the filename.

You can suppress the file name from the output by using the -h option. Conversely, you can also print only the file names with matches by using -l flag.

Feb 26 11:48:19 combo sshd(pam_unix)[6592]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=63.116.211.67  user=root
Feb 26 11:48:19 combo sshd(pam_unix)[6597]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=63.116.211.67  user=root
Feb 26 13:43:51 combo sshd(pam_unix)[6609]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=221.230.128.214  user=nobody

Feb 17 22:40:46 combo ftpd[5272]: connection from 69.177.104.55 (hostname.media.net) at Fri Feb 17 22:40:46 2006