Regular Expressions with Python - 2021

---

---

Python provides support for regular expressions via re module.

Regular expressions are a powerful and standardized way of searching, replacing, and parsing text with complex patterns of characters. There are several good places to look:

1. http://docs.python.org/3.2/library/re.html
2. http://www.regular-expressions.info/
3. https://developers.google.com/edu/python/regular-expressions

1. ^ matches the beginning of a string.
2. $ matches the end of a string.
3. \b matches a word boundary.
4. \d matches any numeric digit.
5. \D matches any non-numeric character.
6. (x|y|z) matches exactly one of x, y or z.
7. (x) in general is a remembered group. We can get the value of what matched by using the groups() method of the object returned by re.search.
8. x? matches an optional x character (in other words, it matches an x zero or one times).
9. x* matches x zero or more times.
10. x+ matches x one or more times.
11. x{m,n} matches an x character at least m times, but not more than n times.
12. ?: matches an expression but do not capture it. Non capturing group.
13. ?= matches a suffix but exclude it from capture. Positive look ahead.
    a(?=b) will match the "a" in "ab", but not the "a" in "ac"
    In other words, a(?=b) matches the "a" which is followed by the string 'b', without consuming what follows the a.
14. ?! matches if suffix is absent. Negative look ahead.
    a(?!b) will match the "a" in "ac", but not the "a" in "ab"
15. ?<= positive look behind
16. ?<! negative look behind

Here is a very simple code replacing ip address:

import re
pattern = re.compile(r'\b(?:[0-9]{1,3}\.){3}[0-9]{1,3}\b')
old = 'IPs : 173.254.28.78 or 167.81.178.97'

new_ip = '127.0.0.1'

replaced = re.sub(pattern, new_ip, old)

print('replaced = %s' %(replaced))

Output:

replaced = IPs : 127.0.0.1 or 127.0.0.1

The re.sub() function performs regular expression-based string substitutions.

>>> import re
>>> re.search('[abc]', 'Space')
<_sre.SRE_Match object at 0x03028C60>
>>> 
>>> re.sub('[abc]', 'o', 'Space')
'Spooe'
>>> re.sub('[aeu]', 'n', re.sub('[abc]', 'o', 'Space'))
'Spoon'

1. Does the string Space contain a, b, or c? Yes, it contains a and c.
2. OK, now find a, b, or c, and replace it with o. Space becomes Spooe.
3. Let's take the output and use it as an input to replace a, e, or u with n.
4. As a result, the Space turned into a Spoon.

In this section, we will learn some of the regular expressions while doing pluralization of nouns.

>>> def pluralize(noun):          
    if re.search('[sxz]$', noun):     
        return re.sub('$', 'es', noun)
    elif re.search('[^aeioudgkprt]h$', noun):
        return re.sub('$', 'es', noun)       
    elif re.search('[^aeiou]y$', noun):      
        return re.sub('y$', 'ies', noun)     
    else:
        return noun + 's'

The four branches of if statements are the implementation of the following four rules of pluralization described below:

1. If a word ends in s, x, or z, add es. Bass becomes basses, fax becomes faxes, and waltz becomes waltzes.
2. If a word ends in a noisy h, add es; if it ends in a silent h, just add s. What's a noisy h? One that gets combined with other letters to make a sound that we can hear. So coach becomes coaches and rash becomes rashes, because we can hear the ch and sh sounds when we say them. But cheetah becomes cheetahs, because the h is silent.
3. If a word ends in y that sounds like i, change the y to ies; if the y is combined with a vowel to sound like something else, just add s. So vacancy becomes vacancies, but day becomes days.
4. If all else fails, just add s.

OK, now is the time to look the code in the example above:

    if re.search('[sxz]$', noun):     
        return re.sub('$', 'es', noun)

The square brackets [] mean match exactly one of these characters. So [sxz] means s, or x, or z, but only one of them. The $ matches the end of string. Combined, this regular expression tests whether noun ends with s, x, or z.

Here, we're replacing the end of the string matched by $ with the string es. In other words, adding es to the string. We could accomplish the same thing with string concatenation, for example noun + 'es', but we opted to use regular expressions for each rule.

    elif re.search('[^aeioudgkprt]h$', noun):
        return re.sub('$', 'es', noun) 

This is another new variation. The ^ as the first character inside the square brackets means something special: negation. [^abc] means any single character except a, b, or c. So [^aeioudgkprt] means any character except a, e, i, o, u, d, g, k, p, r, or t. Then that character needs to be followed by h, followed by end of string. We're looking for words that end in h where the h can be heard.

    elif re.search('[^aeiou]y$', noun):      
        return re.sub('y$', 'ies', noun)  

Same pattern here: match words that end in y, where the character before the y is not a, e, i, o, or u. We're looking for words that end in y that sounds like i.

Let's do some practice with regular expressions:

>>> re.search('[^aeiou]y$', 'emergency')
<_sre.SRE_Match object at 0x03028C98>
>>> re.search('[^aeiou]y$', 'toy')
>>> 
>>> re.search('[^aeiou]y$', 'gay')
>>> 
>>> re.search('[^aeiou]y$', 'taco')

emergency matches this regular expression, because it ends in cy, and c is not a, e, i, o, or u.
toy does not match, because it ends in oy, and we specifically said that the character before the y could not be o. gay does not match, because it ends in ay.
taco does not match, because it does not end in y.

>>> re.search('[^aeiou]y$', 'emergency')
<_sre.SRE_Match object at 0x03028C98>
>>> re.search('[^aeiou]y$', 'toy')
>>> 
>>> re.search('[^aeiou]y$', 'gay')
>>> 
>>> re.search('[^aeiou]y$', 'taco')

Another example:

>>> re.sub('y$', 'ies', 'emergency')
'emergencies'
>>> re.sub('y$', 'ies', 'semitransparency')
'semitransparencies'

This regular expression turns emergency into emergencies and semitransparency into semitransparencies, which is what we wanted. Note that it would also turn toy into toies, but that will never happen in the function because we did that re.search first to find out whether we should do this re.sub.

>>> re.sub('([^aeiou])y$', r'\1ies', 'emergency')
'emergencies'

It is possible to combine these two regular expressions (one to find out if the rule applies, and another to actually apply it) into a single regular expression. Here's what that would look like. We're using a remembered group. The group is used to remember the character before the letter y. Then in the substitution string, we use a new syntax, \1, which means hey, that first group we remembered? put it right here. In this case, we remember the c before the y; when we do the substitution, we substitute c in place of c, and ies in place of y. (If we have more than one remembered group, we can use \2 and \3 and so on.)

Here are the combinations of possible phone numbers that we have to parse. We should be able to get the area code 415, the trunk 867, and the rest of the phone number 5309. We also need to know the extension if any.

1. 415-867-5309
2. 415 867 5309
3. 415.867.5309
4. (415) 867-5309
5. 1-415-867-5309
6. 415-867-5309-9999
7. 415-867-5309x9999
8. 415-867-5309 ext. 9999
9. emergency 1-(415) 867.5309 #9999

This one matches the beginning (^) of the string, and then (\d{3}).
What's \d{3}?
\d means any numeric digit (0 through 9). The {3} means match exactly three numeric digits. It's a variation on the {m,n} syntax which matches an x character at least m times, but not more than n times.
Putting it all in parentheses means match exactly three numeric digits, and then remember them as a group that I can ask for later. Then match a literal hyphen. Then match another group of exactly three digits. Then another literal hyphen. Then another group of exactly four digits. Then match the end of the string.

>>> import re
>>> pattern = re.compile(r'^(\d{3})-(\d{3})-(\d{4})$')
>>> pattern.search('415-867-5309')
<_sre.SRE_Match object at 0x02FCDD40>
>>> pattern.search('415-867-5309').groups()
('415', '867', '5309')

The line

pattern = re.compile(regular expression)

compiles a regular expression into a regular expression object, which can be used for matching using its match() or search() methods. The compiled version, re.compile(), will be cached. In other words, it saves the resulting regular expression object for reuse which makes it efficient when the expression will be used several times in a single program.

For more info on compile, visit Built-in compile.

Also, note that we used raw string (r') in the line

re.compile(r'^(\d{3})-(\d{3})-(\d{4})$')

This is to work around so called the backslash plague. The letter r tells Python that nothing in this string should be escaped; '\t' is a tab character, but r'\t' is really the backslash character \ followed by the letter t. We are better off to use raw strings when dealing with regular expressions; otherwise, things get too confusing too quickly. As we know, regular expressions are confusing enough already.

To get access to the groups that the regular expression parser remembered along the way, we used the groups() method on the MatchObject that the search() method returns. It will return a tuple of however many groups were defined in the regular expression. In this case, we defined three groups, one with three digits, one with three digits, and one with four digits.

So far, it seems to be working fine. However, it breaks at the following lines:

>>> pattern.search('415-867-5309-9999')
>>> pattern.search('415-867-5309-9999').groups()
Traceback (most recent call last):
  File "<pyshell#116>", line 1, in 
    pattern.search('415-867-5309-9999').groups()
AttributeError: 'NoneType' object has no attribute 'groups'

As we see, it doesn't handle a phone number with an extension on the end. For that, we'll need to expand the regular expression. And this is why we should never chain the search() and groups() methods in production code. If the search() method returns no matches, it returns None, not a regular expression match object. Calling None.groups() raises a perfectly obvious exception: None doesn't have a groups() method.

Let's modify the regular expression a little bit;

>>> pattern = re.compile(r'^(\d{3})-(\d{3})-(\d{4})-(\d+)$')
>>> pattern.search('415-867-5309-9875').groups()
('415', '867', '5309', '9875')

This regular expression is almost identical to the previous one. Just as we did before, we match the beginning of the string, then a remembered group of three digits, then a hyphen, then a remembered group of three digits, then a hyphen, then a remembered group of four digits. What's new is that we then match another hyphen, and a remembered group of one or more digits (note that we added \d+), then the end of the string. So, the groups() method now returns a tuple of four elements, since the regular expression now defines four groups to remember.

Though it seems to be working, it breaks again as shown below:

>>> pattern.search('415 867 5309 9999')
>>> 
>>> pattern.search('415-867-5309')
>>> 

Unfortunately, this regular expression is not the final answer either, because it assumes that the different parts of the phone number are separated by hyphens. What if they're separated by spaces, or commas, or dots? We need a more general solution to match several different types of separators. Not only does this regular expression not do everything we want, it's actually a step backwards, because now we can't parse phone numbers without an extension. That's not what we wanted at all; if the extension is there, we want to know what it is, but if it's not there, we still want to know what the different parts of the main number are.

OK, then another regular expression:

>>> pattern = re.compile(r'^(\d{3})\D+(\d{3})\D+(\d{4})\D+(\d+)$')
>>> pattern.search('415 867 5309 9999').groups()
('415', '867', '5309', '9999')
>>> pattern.search('415-867-5309-9999').groups()
('415', '867', '5309', '9999')
>>> 

We're matching the beginning of the string, then a group of three digits, then \D+. What is this doing? Well, \D matches any character except a numeric digit, and + means 1 or more. So \D+ matches one or more characters that are not digits. This is what we're using instead of a literal hyphen, to try to match different separators.

Using \D+ instead of - means we can now match phone numbers where the parts are separated by spaces instead of hyphens. Of course, phone numbers separated by hyphens still work too.

However, this is still not the final answer, because it assumes that there is a separator at all. What if the phone number is entered without any spaces or hyphens at all? It fails:

>>> pattern.search('41586753099999')
>>> 

The regular expression fails even the easier one we've been passed before:

>>> pattern.search('415-867-5309')
>>> 

We have two problems with the regular expression in the previous section, but we can solve both of them with the same technique:

>>> pattern = re.compile(r'^(\d{3})\D*(\d{3})\D*(\d{4})\D*(\d*)$')
>>> pattern.search('41586753099999').groups()
('415', '867', '5309', '9999')
>>> pattern.search('415.867.5309 x9999').groups()
('415', '867', '5309', '9999')
>>> pattern.search('415-867-5309').groups()
('415', '867', '5309', '')

The only change we've made since that last step is changing all the + to *. Instead of \D+ between the parts of the phone number, we now match on \D*. Remember that + means 1 or more? Well, * means zero or more. So now we should be able to parse phone numbers even when there is no separator character at all.

It actually works.
Why?
We matched the beginning of the string, then a remembered group of three digits (415), then zero non-numeric characters, then a remembered group of three digits (867), then zero non-numeric characters, then a remembered group of four digits (5309), then zero non-numeric characters, then a remembered group of an arbitrary number of digits (9999), then the end of the string. Other variations work now too: dots instead of hyphens, and both a space and an x before the extension.

However, we're not finished yet:

>>> pattern.search('(415)8675309 x9999')
>>> 

What's the problem here? There's an extra character '(' before the area code, but the regular expression assumes that the area code is the first thing at the beginning of the string. No problem, we can use the same technique of zero or more non-numeric characters to skip over the leading characters before the area code.

What regular expression we're going to use to parse the extra character before the area code?

>>> pattern = re.compile(r'^\D*(\d{3})\D*(\d{3})\D*(\d{4})\D*(\d*)$')
>>> pattern.search('(415)8675309 x9999').groups()
('415', '867', '5309', '9999')
>>> pattern.search('415-867-5309').groups()
('415', '867', '5309', '')

This is the same as in the previous example, except now we're matching \D*, zero or more non-numeric characters, before the first remembered group (the area code). Notice that we're not remembering these non-numeric characters (they're not in parentheses). If we find them, we'll just skip over them and then start remembering the area code whenever we get to it. We can successfully parse the phone number, even with the leading left parenthesis before the area code. (The right parenthesis after the area code is already handled; it's treated as a non-numeric separator and matched by the \D* after the first remembered group.)

Since the leading characters are entirely optional, this matches the beginning of the string, then zero non-numeric characters, then a remembered group of three digits (415), then one non-numeric character (the hyphen), then a remembered group of three digits (867), then one non-numeric character (the hyphen), then a remembered group of four digits (5309), then zero non-numeric characters, then a remembered group of zero digits, then the end of the string.

Again, here is a phone number we fail:

>>> pattern.search('emergency 1-(415) 867.5309 #9999')
>>> 

Why doesn't this phone number match? That's because we assumed that all the leading characters before the area code were non-numeric characters by putting \D* but there's a 1 before the area code.

So far, the regular expressions have all matched from the beginning of the string. But now we see that there may be an indeterminate amount of stuff at the beginning of the string that we want to ignore. Rather than trying to match it all just so we can skip over it, let's take a different approach:
don't explicitly match the beginning of the string at all.

>>> pattern = re.compile(r'(\d{3})\D*(\d{3})\D*(\d{4})\D*(\d*)$')
>>> pattern.search('emergency 1-(415) 867.5309 #9999').groups()
('415', '867', '5309', '9999')
>>> pattern.search('415-867-5309').groups()
('415', '867', '5309', '')
>>> pattern.search('(415)86753099999').groups()
('415', '867', '5309', '9999')

Note the lack of ^ in this regular expression. We are not matching the beginning of the string anymore. There's nothing that says we need to match the entire input with our regular expression. The regular expression engine will do the hard work of figuring out where the input string starts to match, and go from there.

Now we can successfully parse a phone number that includes leading characters and a leading digit, plus any number of any kind of separators around each part of the phone number.

As we realized, regular expressions are difficult to read, and even if we figure out what one does, there's no guarantee that we'll be able to understand it later. So, what we really need is inline documentation.

Python allows us to do this with something called verbose regular expressions. A verbose regular expression is different from a compact regular expression in two ways:

1. Whitespace is ignored. Spaces, tabs, and carriage returns are not matched as spaces, tabs, and carriage returns. They're not matched at all. If we want to match a space in a verbose regular expression, we'll need to escape it by putting a backslash in front of it)
2. Comments are ignored. A comment in a verbose regular expression is just like a comment in Python code: it starts with a # character and goes until the end of the line. In this case it's a comment within a multi-line string instead of within our source code, but it works the same way.

>>> pattern = re.compile(r'''
                # don't match beginning of string, number can start anywhere
    (\d{3})     # area code is 3 digits (e.g. '415')
    \D*         # optional separator is any number of non-digits
    (\d{3})     # trunk is 3 digits (e.g. '867')
    \D*         # optional separator
    (\d{4})     # rest of number is 4 digits (e.g. '5309')
    \D*         # optional separator
    (\d*)       # extension is optional and can be any number of digits
    $           # end of string
    ''', re.VERBOSE)
>>> pattern.search('emergency 1-(415) 867.5309 #9999').groups()
('415', '867', '5309', '9999')
>>> pattern.search('415-867-5309').groups()
('415', '867', '5309', '')
>>> pattern.search('(415)86753099999').groups()
('415', '867', '5309', '9999')
>>> 

The most important thing to remember when using verbose regular expressions is that we need to pass an extra argument when working with them: re.VERBOSE is a constant defined in the re module that signals that the pattern should be treated as a verbose regular expression. As we can see, this pattern has quite a bit of whitespace (all of which is ignored), and several comments (all of which are ignored). Once we ignore the whitespace and the comments, this is exactly the same regular expression as we saw in our previous sections, but it's a lot more readable.

The following example shows:

1. Recursive file /directory searching
2. Replacing a string in each file using regular expression
3. In listFiles(), list of sub directories is constructed if it's a directory. Otherwise, file extension is checked. Then, the fullpath of the file will be passed to string_replace() function.
4. In the string_replace() function, two files are opened: the file which is passed from listFiles(), and the other one is a new file to write line by line from the input file. After finishing the writing, old file will be removed and then the new file will be renamed with the same name that has been passed.

Actually, I want to replace 2013 with 2014 in my web pages which are under various directories:

<title>Python Tutorial: Regular Expressions with Python - 2013</title>
...
<div style="padding: 10px;">
<span class="titletext">Regular Expressions with Python - 2013    <g:plusone></g:plusone>
</span></div>
...

Here is the python code:

import re
import os

def string_replace(fname):

   pattern1 = '<title>\D*\d*\D*\d*2013\D*\d*</title>'
   pattern2 = '2013\D*\d*<g:plusone>'

   fname_new = 'temp'
   f = open(fname, 'r')
   fnew = open(fname_new, 'w')

   for string in f:
      if(re.search(pattern1, string) or re.search(pattern2, string)):
         newline = re.sub('2013', '2014', string)
         fnew.write(newline)
	 print fname, ':' , newline
      else:
         fnew.write(string)

   f.close()
   fnew.close()
   os.remove(fname)
   os.rename(fname_new, fname)


def listFiles(dir):
    basedir = dir
    print 'current basedir=', basedir
    print "Files in ", os.path.abspath(dir), ": "
    subdirlist = []
    for item in os.listdir(dir):
	#print 'checking if item is a file or a dir', item
	fullpath = os.path.join(basedir,item)
        if os.path.isdir(fullpath):
            print  'dir item=',fullpath
	    subdirlist.append(fullpath)
        else:
	    print 'file item=',fullpath
	    if(item.endswith('.php') or item.endswith('.html')):
	       # print 'file to be processed ********', item
	       print 'file to be processed = ', fullpath
	       string_replace(fullpath)

            
    print '------ new dir ----'
    print 'subdirlist =', subdirlist
    for subdir in subdirlist:
        listFiles(subdir)

# call this to repalce "2013" with "2014" in *.php or *.html
listFiles('/www_root_directory/');