class: center, middle .center[![Python](http://m1.paperblog.com/i/201/2016454/guia-python-conceptos-programacion-atributos--L-DTucOw.png)] # Python Course: Regular Expression Shahid Beheshti University Instructor: S. M. Masoud Sadrnezhaad --- Regular Expression ================ .center[![xkcd #208](xkcd-208-regular_expressions.png)] Source: [xkcd #208](http://xkcd.com/208) --- Regular Expression (Contd) ================ - Also called REs, or regexes, or regex patterns - A tiny, highly specialized **programming language** embedded inside Python and made available through the `re` module. - Using this little language, you specify the rules for the set of possible **strings that you want to match**; this set might contain English sentences, or e-mail addresses, or TeX commands, or anything you like. - You can then ask questions such as "Does this string match the pattern?", or "Is there a match for the pattern anywhere in this string?". - You can also use REs to modify a string or to split it apart in various ways. - You could also try [JavaScript, Python, PHP, and PCRE (regex101)](https://regex101.com). It is an online regex tester and debugger. --- Regular Expression (Contd) ================ - The regular expression language is relatively small and restricted, so **not all possible string processing tasks** can be done using regular expressions. - There are also tasks that *can* be done with regular expressions, but the expressions turn out to be **very complicated**. - In these cases, you may be better off **writing Python code** to do the processing; while Python code will be **slower** than an elaborate regular expression, it will also probably be **more understandable**. - For a detailed explanation of the **computer science** underlying regular expressions (deterministic and non-deterministic finite automata), you can refer to almost any textbook on writing compilers. ![FSM](finite-state-machine.png) --- Matching Characters ================ - Most letters and characters will **simply match themselves**. - For example, the regular expression `test` will match the string `test` exactly. - You can enable a **case-insensitive mode** that would let this RE match `Test` or `TEST` as well; more about this later. - There are exceptions to this rule; some characters are special `metacharacters`, and **don't match themselves**. - Instead, they signal that some out-of-the-ordinary thing should be matched, or they affect other portions of the RE by **repeating them or changing their meaning**. - Here's a complete list of the metacharacters; their meanings will be discussed in the rest of this section. ```regex . ^ $ * + ? { } [ ] \ | ( ) ``` --- Metacharacters ================ - The first metacharacters we'll look at are `[` and `]`. - They're used for specifying a **character class**, which is a set of characters that you wish to match. Characters can be listed individually, or a **range of characters** can be indicated by giving two characters and separating them by a `'-'`. - For example, `[abc]` will match any of the characters `a`, `b`, or `c`; this is the same as `[a-c]`, which uses a range to express the same set of characters. - If you wanted to match only lowercase letters, your RE would be `[a-z]`. - Metacharacters are **not active inside classes**. - For example, `[akm$]` will match any of the characters `'a'`, `'k'`, `'m'`, or `'$'`; `'$'` is usually a metacharacter, but inside a character class it's stripped of its special nature. --- Metacharacters (Contd) ================ - You can match the **characters not listed** within the class by `complementing` the set. - This is indicated by including a `'^'` as the first character of the class; `'^'` outside a character class will simply match the `'^'` character. - For example, `[^5]` will match any character except `'5'`. - Perhaps the most important metacharacter is the backslash, `\`. - As in Python string literals, the backslash can be **followed by various characters** to signal various special sequences. - It's also used to **escape all the metacharacters** so you can still match them in patterns; for example, if you need to match a `[` or `\`, you can precede them with a backslash to remove their special meaning: `\[` or `\\`. --- Metacharacters (Contd) ================ - Some of the special sequences beginning with `'\'` represent **predefined sets of characters** that are often useful. - `\d` Matches any decimal digit; this is equivalent to the class `[0-9]`. - `\D` Matches any non-digit character; this is equivalent to the class `[^0-9]`. - `\s` Matches any whitespace character; this is equivalent to the class `[ \t\n\r\f\v]`. - `\S` Matches any non-whitespace character; this is equivalent to the class `[^ \t\n\r\f\v]`. - `\w` Matches any alphanumeric character; this is equivalent to the class `[a-zA-Z0-9]`. - `\W` Matches any non-alphanumeric character; this is equivalent to the class `[^a-zA-Z0-9]`. --- Metacharacters (Contd) ================ - These sequences can be included inside a character class. For example, `[\s,.]` is a character class that will match any whitespace character, or `','` or `'.'`. - The final metacharacter in this section is `.`. It matches anything except a newline character, and there's an alternate mode (`re.DOTALL`) where it will match even a newline. `.` is often used where you want to match "any character". --- Repeating Things ================ - Another capability is that you can specify that portions of the RE must be repeated a certain number of times. - The first metacharacter for **repeating things** that we'll look at is `*`. - `*` doesn't match the literal character `'*'`; instead, it specifies that the **previous character** can be matched **zero or more times**, instead of exactly once. - For example, `ca*t` will match `'ct'` (0 `'a'` characters), `'cat'` (1 `'a'`), `'caaat'` (3 `'a'` characters), and so forth. - A step-by-step **example** will make this more obvious. Let's consider the expression `a[bcd]*b`. This matches the letter `'a'`, zero or more letters from the class `[bcd]`, and finally ends with a `'b'`. Now imagine matching this RE against the string `'abcbd'`. --- Repeating Things (Contd) ================ ```restructuredtext +------+-----------+---------------------------------+ | Step | Matched | Explanation | +======+===========+=================================+ | 1 | `a` | The `a` in the RE matches. | +------+-----------+---------------------------------+ | 2 | `abcbd` | The engine matches `[bcd]*`, | | | | going as far as it can, which | | | | is to the end of the string. | +------+-----------+---------------------------------+ | 3 | *Failure* | The engine tries to match | | | | `b`, but the current position | | | | is at the end of the string, so | | | | it fails. | ... ``` --- Repeating Things (Contd) ================ ```restructuredtext ... | 4 | `abcb` | Back up, so that `[bcd]*` | | | | matches one less character. | +------+-----------+---------------------------------+ | 5 | *Failure* | Try `b` again, but the | | | | current position is at the last | | | | character, which is a `'d'`. | +------+-----------+---------------------------------+ | 6 | `abc` | Back up again, so that | | | | `[bcd]*` is only matching | | | | `bc`. | +------+-----------+---------------------------------+ | 6 | `abcb` | Try `b` again. This time | | | | the character at the | | | | current position is `'b'`, so | | | | it succeeds. | +------+-----------+---------------------------------+ ``` The end of the RE has now been reached, and it has matched `'abcb'`. This demonstrates how the matching engine goes as far as it can at first, and if no match is found it will then progressively back up and retry the rest of the RE again and again. --- Repeating Things (Contd) ================ - Another repeating metacharacter is `+`, which matches one or more times. - Pay careful attention to the difference between `*` and `+`; `*` matches *zero* or more times, so whatever's being repeated may not be present at all, while `+` requires at least *one* occurrence. - To use a similar example, `ca+t` will match `'cat'` (1 `'a'`), `'caaat'` (3 `'a'`\ s), but won't match `'ct'`. - The question mark character, `?`, matches either **once or zero times**; you can think of it as marking something as being **optional**. - For example, `home-?brew` matches either `'homebrew'` or `'home-brew'`. --- Repeating Things (Contd) ================ - The most complicated repeated qualifier is `{m,n}`, where *m* and *n* are decimal integers. This qualifier means there must be at least *m* repetitions, and at most *n*. - For example, `a/{1,3}b` will match `'a/b'`, `'a//b'`, and `'a///b'`. It won't match `'ab'`, which has no slashes, or `'a////b'`, which has four. - You can **omit** either *m* or *n*; in that case, **a reasonable value is assumed** for the missing value. - Omitting *m* is interpreted as a **lower limit of 0**, while omitting *n* results in an **upper bound of infinity**. - Three other qualifiers can all be expressed using this notation. `{0,}` is the same as `*`, `{1,}` is equivalent to `+`, and `{0,1}` is the same as `?`. - It's better to use `*`, `+`, or `?` when you can, simply because they're **shorter and easier to read.** --- Using Regular Expressions in Python ================ - The `re` module provides an interface to the regular expression engine, allowing you to compile REs into objects and then perform matches with them. Compiling Regular Expressions ----------------------------- - Regular expressions are compiled into **pattern objects**, which have methods for **various operations** such as searching for pattern matches or performing string substitutions. ```python >>> import re >>> p = re.compile('ab*') >>> p re.compile('ab*') ``` - `re.compile` also accepts an **optional *flags* argument**, used to enable various special features and syntax variations. We'll go over the available settings later, but for now a single example will do: ```python >>> p = re.compile('ab*', re.IGNORECASE) ``` --- Compiling Regular Expressions (Contd) ================ - The RE is passed to `re.compile` as a string. - REs are handled as strings because regular expressions aren't part of the core Python language, and no special syntax was created for expressing them. - There are applications that don't need REs at all, so there's no need to bloat the language specification by including them. - Instead, the `re` module is simply a C extension module included with Python, just like the `socket` or `zlib` modules. - Putting REs in strings keeps the Python language simpler, but has one disadvantage. --- Using Regular Expressions in Python ================ Performing Matches ------------------ Once you have an object representing a compiled regular expression, what do you do with it? Pattern objects have **several methods and attributes**. Only the most significant ones will be covered here. ```restructuredtext +------------------+-----------------------------------------------+ | Method/Attribute | Purpose | +==================+===============================================+ | `match()` | Determine if the RE matches at the beginning | | | of the string. | +------------------+-----------------------------------------------+ | `search()` | Scan through a string, looking for any | | | location where this RE matches. | +------------------+-----------------------------------------------+ | `findall()` | Find all substrings where the RE matches, and | | | returns them as a list. | +------------------+-----------------------------------------------+ | `finditer()` | Find all substrings where the RE matches, and | | | returns them as an `iterator`. | +------------------+-----------------------------------------------+ ``` --- More Metacharacters ================ - There are some metacharacters that we haven't covered yet. Most of them will be covered in this section. - Some of the remaining metacharacters to be discussed are `zero-width assertions`. - They don't cause the engine to advance through the string; instead, **they consume no characters at all**, and **simply succeed or fail**. - Zero-width assertions should never be repeated, because if they match once at a given location, they can obviously be matched an infinite number of times. --- More Metacharacters (Contd) ================ - **`|`** - Alternation, or the **"or" operator**. If *A* and *B* are regular expressions, `A|B` will match any string that matches either *A* or *B*. - `|` has very low precedence in order to make it work reasonably when you're alternating multi-character strings. - `Crow|Servo` will match either `'Crow'` or `'Servo'`, not `'Cro'`, a `'w'` or an `'S'`, and `'ervo'`. - To match a literal `'|'`, use `\|`, or enclose it inside a character class, as in `[|]`. --- More Metacharacters (Contd) ================ - **`^`** - Matches at the **beginning of lines**. - For example, if you wish to match the word `From` only at the beginning of a line, the RE to use is `^From`. : ```python >>> print(re.search('^From', 'From Here to Eternity')) #doctest: +ELLIPSIS
>>> print(re.search('^From', 'Reciting From Memory')) None ``` - To match a literal `'^'`, use `\^`. --- More Metacharacters (Contd) ================ - **`$`** - Matches at the end of a line, which is defined as either the **end of the string**, or **any location followed by a newline character**. ```python >>> print(re.search('}$', '{block}')) #doctest: +ELLIPSIS
>>> print(re.search('}$', '{block} ')) None >>> print(re.search('}$', '{block}\n')) #doctest: +ELLIPSIS
``` - To match a literal `'$'`, use `\$` or enclose it inside a character class, as in `[$]`. --- # References * https://docs.python.org/3/howto/regex.html --- class: center, middle .center[![Python](http://m1.paperblog.com/i/201/2016454/guia-python-conceptos-programacion-atributos--L-DTucOw.png)] # Thank you. Any questions?