Introduction Regular expressions called REs, or regexes, or regex patterns are essentially a tiny, highly specialized programming language embedded inside Python and made available through the module. Using this little language, you specify the rules for the set of possible strings that you want to match; this set might contain English sentences, or e-mail addresses, or TeX commands, or anything you like. You can also use REs to modify a string or to split it apart in various ways. Regular expression patterns are compiled into a series of bytecodes which are then executed by a matching engine written in C. For advanced use, it may be necessary to pay careful attention to how the engine will execute a given RE, and write the RE in a certain way in order to produce bytecode that runs faster. The regular expression language is relatively small and restricted, so not all possible string processing tasks can be done using regular expressions. There are also tasks that can be done with regular expressions, but the expressions turn out to be very complicated. In these cases, you may be better off writing Python code to do the processing; while Python code will be slower than an elaborate regular expression, it will also probably be more understandable. For a detailed explanation of the computer science underlying regular expressions deterministic and non-deterministic finite automata , you can refer to almost any textbook on writing compilers. Matching Characters Most letters and characters will simply match themselves. For example, the regular expression test will match the string test exactly. You can enable a case-insensitive mode that would let this RE match Test or TEST as well; more about this later. Instead, they signal that some out-of-the-ordinary thing should be matched, or they affect other portions of the RE by repeating them or changing their meaning. Much of this document is devoted to discussing various metacharacters and what they do. Characters can be listed individually, or a range of characters can be indicated by giving two characters and separating them by a '-'. Metacharacters are not active inside classes. You can match the characters not listed within the class by complementing the set. As in Python string literals, the backslash can be followed by various characters to signal various special sequences. For a complete list of sequences and expanded class definitions for Unicode string patterns, see the last part of in the Standard Library reference. These sequences can be included inside a character class. The final metacharacter in this section is.. Another capability is that you can specify that portions of the RE must be repeated a certain number of times. A step-by-step example will make this more obvious. Now imagine matching this RE against the string 'abcbd'. Step Matched Explanation 1 a The a in the RE matches. This time the character at the current position is 'b', so it succeeds. The end of the RE has now been reached, and it has matched 'abcb'. This demonstrates how the matching engine goes as far as it can at first, and if no match is found it will then progressively back up and retry the rest of the RE again and again. Another repeating metacharacter is +, which matches one or more times. There are two more repeating qualifiers. The question mark character,? The most complicated repeated qualifier is m,n, where m and n are decimal integers. This qualifier means there must be at least m repetitions, and at most n. You can omit either m or n; in that case, a reasonable value is assumed for the missing value. Omitting m is interpreted as a lower limit of 0, while omitting n results in an upper bound of infinity. Readers of a reductionist bent may notice that the three other qualifiers can all be expressed using this notation. IGNORECASE The RE is passed to as a string. Instead, the module is simply a C extension module included with Python, just like the or modules. Putting REs in strings keeps the Python language simpler, but has one disadvantage which is the topic of the next section. To figure out what to write in the program code, start with the desired string to be matched. However, to express this as a Python string literal, both backslashes must be escaped again. In REs that feature backslashes repeatedly, this leads to lots of repeated backslashes and makes the resulting strings difficult to understand. Regular expressions will often be written in Python code using this raw string notation. Pattern objects have several methods and attributes. Only the most significant ones will be covered here; consult the docs for a complete listing. You can learn about this by interactively experimenting with the module. If you have available, you may also want to look at , a demonstration program included with the Python distribution. It allows you to enter REs and strings, and displays whether the RE matches or fails. This HOWTO uses the standard Python interpreter for its examples. You can explicitly print the result of match to make this clear. Since the method only checks if the RE matches at the start of a string, start will always be zero. However, the method of patterns scans through the string, so the match may not start at zero in that case. These functions take the same arguments as the corresponding pattern method with the RE string added as the first argument, and still return either None or a instance. Should you use these module-level functions, or should you get the pattern and call its methods yourself? Compilation Flags Compilation flags let you modify some aspects of how regular expressions work. Flags are available in the module under two names, a long name such as IGNORECASE and a short, one-letter form such as I. Multiple flags can be specified by bitwise OR-ing them; re. M sets both the I and M flags, for example. IGNORECASE, I Do case-insensitive matches. LOCALE, L Do a locale-aware match. I IGNORECASE Perform case-insensitive matching; character class and literal strings will match letters by ignoring case. Full Unicode matching also works unless the ASCII flag is used to disable non-ASCII matches. Spam will match 'Spam', 'spam', 'spAM', or 'ſpam' the latter is matched only in Unicode mode. Locales are a feature of the C library intended to help in writing programs that take account of language differences. If your system is configured properly and a French locale is selected, certain C functions will tell the program that the byte corresponding to é should also be considered a letter. S DOTALL Makes the '. This is only meaningful for Unicode patterns, and is ignored for byte patterns. X VERBOSE This flag allows you to write regular expressions that are more readable by granting you more flexibility in how you can format them. When this flag has been specified, whitespace within the RE string is ignored, except when the whitespace is in a character class or preceded by an unescaped backslash; this lets you organize and indent the RE more clearly. Most of them will be covered in this section. Some of the remaining metacharacters to be discussed are zero-width assertions. This means that zero-width assertions should never be repeated, because if they match once at a given location, they can obviously be matched an infinite number of times. If A and B are regular expressions, A B will match any string that matches either A or B. Crow Servo will match either 'Crow' or 'Servo', not 'Cro', a 'w' or an 'S', and 'ervo'. Unless the MULTILINE flag has been set, this will only match at the beginning of the string. In MULTILINE mode, this also matches immediately after each newline within the string. This is a zero-width assertion that matches only at the beginning or end of a word. A word is defined as a sequence of alphanumeric characters, so the end of a word is indicated by whitespace or a non-alphanumeric character. The following example looks the same as our previous RE, but omits the 'r' in front of the RE string. Grouping Frequently you need to obtain more information than just whether the RE matched or not. Regular expressions are often used to dissect strings by writing a RE divided into several subgroups which match different components of interest. For example, an RFC-822 header line is divided into a header name and a value, separated by a ':', like this: From: author example. Groups are marked by the ' ', ' ' metacharacters. Groups are numbered starting with 0. For example, the following RE detects doubled words in a string. Non-capturing and Named Groups Elaborate REs may use many groups, both to capture substrings of interest, and to group and structure the RE itself. In complex REs, it becomes difficult to keep track of the group numbers. There are two features which help with this problem. Perl 5 is well known for its powerful additions to standard regular expressions. The solution chosen by the Perl developers was to use?... The characters immediately after the? You can make this fact explicit by using a non-capturing group:? A more significant feature is named groups: instead of referring to them by numbers, groups can be referenced by a name. The syntax for a named group is one of the Python-specific extensions:? Named groups behave exactly like capturing groups, and additionally associate a name with a group. The syntax for backreferences in an expression such as... This is another Python extension:? Lookahead assertions are available in both positive and negative form, and look like this:? This succeeds if the contained regular expression, represented here by... Consider a simple pattern to match a filename and split it apart into a base name and an extension, separated by a.. For example, in news. The pattern to match this is quite simple:. This regular expression matches foo. Now, consider complicating the problem a bit; what if you want to match filenames where the extension is not bat? Worse, if the problem changes and you want to exclude both bat and exe as extensions, the pattern would get even more complicated and confusing. A negative lookahead cuts through all this confusion:. Excluding another filename extension is now easy; simply add it as an alternative inside the assertion. The following pattern excludes filenames that end in either bat or exe:. If capturing parentheses are used in the RE, then their contents will also be returned as part of the resulting list. If maxsplit is nonzero, at most maxsplit splits are performed. You can limit the number of splits made, by passing a value for maxsplit. When maxsplit is nonzero, at most maxsplit splits will be made, and the remainder of the string is returned as the final element of the list. In the following example, the delimiter is any sequence of non-alphanumeric characters. If capturing parentheses are used in the RE, then their values are also returned as part of the list. Search and Replace Another common task is to find all the matches for a pattern, and replace them with a different string. The method takes a replacement value, which can be either a string or a function, and the string to be processed. The optional argument count is the maximum number of pattern occurrences to be replaced; count must be a non-negative integer. The default value of 0 means to replace all occurrences. This lets you incorporate portions of the original text in the resulting replacement string. The following substitutions are all equivalent, but use all three variations of the replacement string. If replacement is a function, the function is called for every non-overlapping occurrence of pattern. On each call, the function is passed a argument for the match and can use this information to compute the desired replacement string and return it. The pattern may be provided as an object or as a string; if you need to specify regular expression flags, you must either use a pattern object as the first parameter, or use embedded modifiers in the pattern string, e. Use String Methods Sometimes using the module is a mistake. One example might be replacing a single fixed string with another one; for example, you might replace word with deed. Note that replace will also replace word inside words, turning swordfish into sdeedfish, but the naive RE word would have done that, too. Another common task is deleting every occurrence of a single character from a string or replacing it with another single character. You might do this with something like re. In short, before turning to the module, consider whether your problem can be solved with a faster and simpler string method. Resist this temptation and use instead. The regular expression compiler does some analysis of REs in order to speed up the process of looking for a match. One such analysis figures out what the first character of a match must be; for example, a pattern starting with Crow must match starting with a 'C'. The analysis lets the engine quickly scan through the string looking for the starting character, only trying the full match if a 'C' is found. Use an HTML or XML parser module for such tasks. REs of moderate complexity can become lengthy collections of backslashes, parentheses, and metacharacters, making them difficult to read and understand. For such REs, specifying the flag when compiling the regular expression can be helpful, because it allows you to format the regular expression more clearly. VERBOSE flag has several effects. In addition, you can also put comments inside a RE; comments extend from a character to the next newline. VERBOSE This is far more readable than: Feedback Regular expressions are a complicated topic. Did this document help you understand them? If so, please send suggestions for improvements to the author. Consider checking it out from your library.