2_regex.Rmd

---
title: |-
  Python Crash Course \
   Part 2 -- Intro to Regex
author: "Rob Colautti"
---


# 1. Overview

Regular Expressions

* aka 'regex' and 'regexp'

* a sort of find-and-replace for nerds

* one of the most powerful data tools I have ever learned

* requires patience and lots of **practice**


# 2. Basic Regex Functions in Python

The `re` module in Python contains functions for regular expressions

## Handy regex functions in `re`

### Searching

`re.match()` - match between _pattern_ and whole _target string_

`re.search()` - find first match of _pattern_ within _target string_

`re.findall()` - find all non-overlapping _pattern_ within _target string_

### Modifying

`re.split()` - split _target string_ according to _pattern_

`re.sub()` - replace occurrences of _pattern_ within _string_


## `re.match()`

Returns 'None' if no match
```{python}
import re
m1=re.match("salicaria","salicaria")
print(m1)
m2=re.match("a","salicaria")
print(m2)
m3=re.match("z","salicaria")
print(m3)
```

## `re.search()`

Contrast output with `re.match()` 
```{python}
import re
m1=re.search("salicaria","salicaria")
print(m1)
m2=re.search("a","salicaria")
print(m2)
m3=re.search("z","salicaria")
print(m3)
```

> What is different?

## `re.findall()`

Note `[]` instead of `None` when no match is found
```{python}
import re
m1=re.findall("salicaria","salicaria")
print(m1)
m2=re.findall("a","salicaria")
print(m2)
m3=re.findall("z","salicaria")
print(m3)
```

## `re.split()`

Compare this output with previous functions
```{python}
import re
m1=re.split("salicaria","salicaria")
print(m1)
m2=re.split("a","salicaria")
print(m2)
m3=re.split("z","salicaria")
print(m3)
```

> What is different?

## `re.sub()`

For this function, we have to add a _replace string_ parameter 
```{python}
import re
m1=re.sub("salicaria","X","salicaria")
print(m1)
m2=re.sub("a","X","salicaria")
print(m2)
m3=re.sub("z","X","salicaria")
print(m3)
```

# 3. Wildcards

## Escape character: `\`

The escape character tells the Python interpreter that the next string is not to be interpreted literally. Unlike regular expressions in R, python uses only a single backslash.

## `\w`

All letters and digits (aka 'words')
```{python}
from re import sub
m1=sub("w","*","...which 1-100 words get replaced?")
print(m1)
m2=sub("\w","*","...which 1-100 words get replaced?")
print(m2)
```

## `\W` 

Upper case `\W` means the inverse of lower-case `\w` (non-word and non-number)
```{python}
m1=sub("\W","*","...which 1-100  words get replaced?")
print(m1)
```

## `\s` 

Empty space
```{python}
m1=sub("\s","*","...which 1-100  words get replaced?")
print(m1)
```

## `\t`

Tab character (useful for tab-delimited data files)
```{python}
m1=sub("\t","*","...which 1-100  words get replaced?")
print(m1)
```

## `\d` 

Digits (numbers)
```{python}
m1=sub("\d","*","...which 1-100  words get replaced?")
print(m1)
```

## `\D`

Inverse of `\d` (non-digits)
```{python}
m1=sub("\D","*","...which 1-100  words get replaced?")
print(m1)
```

## `.` = any character except new line
```{python}
m1=sub(".","*","...which 1-100  words get replaced?")
print(m1)
```

So how to search for a period? Use the escape character
```{python}
m1=sub("\.","*","...which 1-100  words get replaced?")
print(m1)
```

## __Two more special wildcards:__

### `\r`

Carriage return

### `\n` 

Newline character

Unix/Mac files -- lines usually end with `\n` only

Windows/DOS files -- lines usually end with `\r\n`

> FUN FACT: Where does the term 'Carriage return' come from?

# 4. Special characters:

## Or: `|`

The vertical bar or 'pipe' is often located above the backslash (shift-backslash). It is the regex equivalent of __or__

For example, look for w __or__ e
```{python}
m1=sub("w|e","*","...which 1-100  words get replaced?")
print(m1)
```

## `*`, `?`, `+` and `{}`

Use `*`, `?`, `+` and `{}` for more complicated searches

Look at these examples carefully
```{python}
m1=sub("\w","*","...which 1-100 words get replaced?")
print(m1)
```

### `+` 
1 or more occurrences
```{python}
m1=sub("\w+","*","...which 1-100 words get replaced?")
print(m1)
```

### `?`

This tells regex to do a 'lazy' search (find the first occurrance)
```{python}
m1=sub("\w?","*","...which 1-100 words get replaced?")
print(m1)
```

### `*` 

This tells regex to do a 'greedy' search (find the last occurrence)
```{python}
m1=sub("\w*","*","...which 1-100 words get replaced?")
print(m1)
```

## `{n,m}` 

Find between n to m matches
```{python}
m1=sub("\w{3,4}","*","...which 1-100 words get replaced?")
print(m1)
```

### `{n}` = exactly n matches
```{python}
m1=sub("\w{3}","*","...which 1-100 words get replaced?")
print(m1)
```

### `{n,}`= n or more matches
```{python}
m1=sub("\w{4,}","*","...which 1-100 words get replaced?")
print(m1)
```

# 5. Multiple search: `[]`

Use square brackets to find any matching characters. 
```{python}
m1=sub("which","*","...which 1-100 words get replaced?")
print(m1)
m2=sub("[which]","*","...which 1-100 words get replaced?")
print(m2)
```

## Range of characters `-` 
Use dash for a range of numbers
```{python}
m1=sub("[1-4]","*","1234567890")
print(m1)
```

or letters
```{python}
m1=sub("[a-z]","*","AaBbCcDd")
print(m1)
m2=sub("[A-Z]","*","AaBbCcDd")
print(m2)
m3=sub("[A-z]","*","AaBbCcDd")
print(m3)
```

# 6. `^`Start and end of line`$`

## `^` Start of line

Find species starting with "s"
```{python}
m1=sub("^s","*","start of lines")
print(m1)
```

## `[^]`

IMPORTANT: ^ Also 'negates' when used within []
Find species containing any letter other than s
```{r}
m1=sub("[^s]","*","start of lines")
print(m1)
```

## $ End of line

Find species ending with "a"
```{python}
m1=sub("s$","*","start of lines")
print(m1)
```

# 7. Capture text: `()`

Capture text using `()` and reprint using `\\1`, `\\2`, etc

Replace each word with its first letter
```{python}
m1=sub(".*(\w\w+).*","\\1","...which 1-100 words get replaced?")
print(m1)
```

Pull out only the numbers and reverse their order
```{python}
m1=sub(".*([0-9]+)-([0-9]+).*","\\2-\\1","...which 1-100 words get replaced?")
print(m1)
```

Reverse first two letters of each 'word' containing 3+ characters
```{python}
m1=sub("(\w)(\w)(\w+)","\\2\\1\\3","...which 1-100 words get replaced?")
print(m1)
```

## `group()` - find matching group

As you can see above, `re.match()` and `re.search()` return an object if there is a match. What if we want to see what was matched? Use `.group()`
```{python}
import re
m1=re.match("salicaria","salicaria")
print(m1.group())
```

But group() produces an error if no match was generated:
```{python, error=T}
import re
m2=re.match("z","salicaria")
print(m2)
print(m1.group())
```

## `groups()` - find matching subgroups

Brackets can be used to subset searches, and then `groups()` identifies the matched groups.
```{python}
import re
m1=re.match("sal","salicaria")
print(m1)
print(m1.groups())

m2=re.match("(s)(a)(l)","salicaria")
print(m2)
print(m2.groups())
print(m2.group(1))
print(m2.group(2))
print(m2.group(3))
```

# PRACTICE EXERCISES

## 1. Consider a vector of email addresses scraped from the internet:

  * robert 'dot' colautti 'at' queensu 'dot' ca
  * chris.eckert[at]queensu.ca
  * lonnie.aarssen at queensu.ca

Use regular expressions to convert all email addresses to the standard format: name@queensu.ca

## 2. Create a random sequence of DNA:
```{python}
MySeq="ATGTGTGATAGATATAGTTTATAG"
```

    * Replace T with U
    * Find all start codons (AUG) and stop codons (UAA, UAG, UGA)
    * Find all open reading frames (hint: consider each sequence beginning with AUG and ending with a stop codon; how do you know if both sequences are in the same reading frame?)
    * Count the length of bp for all open reading frames

## 3. More online examples

[http://regex.sketchengine.co.uk/extra_regexps.html](http://regex.sketchengine.co.uk/extra_regexps.html)

## 4. Regex Golf

Have fun! [LINK](https://alf.nu/RegexGolf)