Following this lecture, you should be able to:
Write simple regular expressions to match patterns in character vectors
Use regular expressions to transform datasets
Big idea: Strings are created with " or '
In R, we create strings by enclosing characters in single (') or double (") quotes. As a useful convention, we default to double quotes.
Note: What is a literal?
In programming, a “literal” is a representation of a specific value type in the code itself. Literals are created using specific notation instead of more conventional function syntax. For example, we create a string literal by enclosing characters in quotes; we don’t use a quote() or string() function.
Literals are the things that we use to build vectors from scratch. We worked with several other examples of R literals during our Vectors, Lists, and Tibbles lecture. For example, you might remember that integer literals are notated using an L (e.g., 1L), and that Boolean literals are notated using the reserved names TRUE and FALSE.
What happens if our string contains double quotes?
Just enclose the string in single quotes!
How about when your string contains both single and double quotes?
In R, " and ' create string literals — this is their primary meaning
We can escape this primary meaning by using the back-slash (\)
The \ says, “ignore the primary meaning of " in favor of its literal meaning.”
To view the underlying representation of this string literal we can use str_view()
[1] │ I think it was Ghandi who once said, "Prof. Masons's lectures are the best!"Wait, but if \ means “escape”, then how do we represent the literal back-slash character?
Well we need to escape the escape of course!
The primary meaning of n is, well, literally the character n
We can escape this primary meaning as \n to mean “new line”
The primary meaning of t is, again, literally the character t
We can escape this primary meaning as \t to mean “tab”
Tip: New lines in Windows
You may come across strings containing \r\n. This is the way that “new line” is encoded on Windows platforms.
You can use the R editor below to complete the following exercises. The editor does not save your work. If you’d prefer, you can also complete exercises in Posit Cloud.
filter() we used equality (==) to subset rows based on the identity of valuesWe can easily imagine many additional characteristics of numbers by which to filter
All numbers divisible by 3
All prime numbers
All odd numbers
All negative numbers
It can, at first, be difficult to come up with a similar list for strings
Once we get started, however, the list of possible string characteristics will seem unending
The length of the string
The case of the characters in the string
The presence of number characters in the string
The presence of special characters in the string
The presence of whitespace in the string
The characters at the start of the string
The characters at the end of the string
The characters not in the string
The number of repeated characters
The order of the characters in the string
Regular expressions (regexs; regexps) give us the ability to describe these (often) complex string patterns in a concise way
Big idea: Pattern matching
We learn how to write regular expressions so that we can find patterns among the characters of strings. Once we’ve described a pattern using a regular expression, we can do all sorts of useful things including:
Detect patterns in strings to filter() using str_detect()
Remove matched characters using str_remove()
Extract matched characters to mutate() using str_extract()
More on these functions at the end of the lecture.
At first (and often for much longer), regular expressions will seem incredibly unintuitive and somewhat magical (🧙♂️)
Here’s an example of a fairly commonly applied type of regular expression
\^(?=.*\[A-Za-z\])(?=.*\d)(?=.\*\[\@$!%*#?&])[A-Za-z\d@$!%\*#?&\]{8,}\$
Any guesses as to what character pattern this matches?
Used to check password requirements — matches to strings with:
At least 8 characters
At least one letter
At least one numbers
At least one special character
Note: The fruit vector
We’ll be using the stringr::fruit character vector throughout the following slides. We’ve met this example data before, but here again is what it looks like:
The simplest patterns to match to are just sequences of literal characters
For these basic patterns, regular expressions look just like strings (phew!)
For example, let’s find all strings from fruit that contain the letter "q"
[43] │ kum<q>uat
[46] │ lo<q>uat
[67] │ <q>uince"berry" in the name [6] │ bil<berry>
[7] │ black<berry>
[10] │ blue<berry>
[11] │ boysen<berry>
[19] │ cloud<berry>
[21] │ cran<berry>
[29] │ elder<berry>
[32] │ goji <berry>
[33] │ goose<berry>
[38] │ huckle<berry>
[50] │ mul<berry>
[70] │ rasp<berry>
[73] │ salal <berry>
[76] │ straw<berry>In the last slide, we wrote the following regular expression: "berry"
Let’s track "berry" on its journey from string literal in R code to the character pattern it matches to
| Stage | Representation | |
|---|---|---|
| 1 | String literal in R code | "berry" |
| 2 | String value/regular expression | berry |
| 3 | Match | berry |
We give R the literal string "berry"
This literal is represented as the string value berry
Which is interpreted as the regular expression berry
Which is then evaluated to match the character pattern berry
All of the characters in "berry" are literals — they match to exactly the characters that they are
Certain other characters, called metacharacters, have non-literal meanings
To illustrate, let’s find all fruit names starting with “a”
^ is a metacharacter that anchors a pattern at the start of a string
We can find all names that end in "n" using a different anchor, $
Tip: Remembering ^ and $
Here’s a useful mnemonic to help you remember that ^ (the exponentiation or “power” symbol) is the start anchor, and $ the end: if you start with power, you end with money. A bleak but accurate commentary on our world…
| Stage | Representation | |
|---|---|---|
| 1 | String literal in R code | "n$" |
| 2 | String value/regular expression | n$ |
| 3 | Match | n at the end |
We’ve seen that we can use \ to escape primary meanings in string-land
How do we escape in regexp-land, and why would we need to?
$ character$ is a metacharacter — it’ll match to end of all strings| Stage | Representation | |
|---|---|---|
| 1 | String literal in R code | "$" |
| 2 | String value\regular expression | $ |
| 3 | Match | the end of the string |
$’s primary meaning, let’s give \ a try!Error: '\$' is an unrecognized escape in character string (<text>:1:30)Error?! Remember, we go from R-land (string literal) to string-land
In string-land, $ has no secondary meaning to escape into, so we get an error
We need to escape the escape!
| Stage | Representation | |
|---|---|---|
| 1 | String literal in R code | "\\$" |
| 2 | String value/regular expression | \$ |
| 3 | Match | $ |
The . metacharacter matches to all characters except the newline character \n
Find all fruit names that end with a any character e
. character, we’d need to double escapeYou can use the R editor below to complete the following exercises. The editor does not save your work. If you’d prefer, you can also complete exercises in Posit Cloud.
fruit that start with bl^, $, and .), find all the fruits that are exactly seven characters longnumbers vector created above, find all decimal place characters"\\\\" match to?# exercise 2
str_view(fruit, pattern = "^bl")
# exercise 3
str_view(fruit, pattern = "^.......$")
# exercise 4
str_view(numbers, pattern = "\\.")
# exercise 5
# it matches to the literal \ character
# "\\\\" becomes \\ (two \ get escaped in string-land)
# and \\ (an escaped escape in regexp-land) matches to \
str_view(string = "\\", pattern = "\\\\") [2] │ <apricot>
[3] │ <avocado>
[5] │ <bell pepper>
[6] │ <bilberry>
[7] │ <blackberry>
[8] │ <blackcurrant>
[9] │ <blood orange>
[10] │ <blueberry>
[11] │ <boysenberry>
[12] │ <breadfruit>
[13] │ <canary melon>
[14] │ <cantaloupe>
[15] │ <cherimoya>
[17] │ <chili pepper>
[18] │ <clementine>
[19] │ <cloudberry>
[20] │ <coconut>
[21] │ <cranberry>
[22] │ <cucumber>
[23] │ <currant>
... and 32 moreNote: The words and sentences vectors
The {stringr} package includes two other example data objects called words and sentences.
There are 980 word strings[1] "a" "able" "about" "absolute" "accept" There are 720 sentence strings[1] "The birch canoe slid on the smooth planks."
[2] "Glue the sheet to the dark blue background."
[3] "It's easy to tell the depth of a well."
[4] "These days a chicken leg is a rare dish."
[5] "Rice is often served in round bowls." Make a pattern optional (zero or one) with ?
Allow a pattern to repeat one or more times with +
Make a pattern optional or allow it to repeat one or more times with *
[11] │ The <boy> was there when the sun rose.
[25] │ The beauty of the view stunned the young <boy>.
[423] │ The <boy> owed his pal thirty cents.
[591] │ Stop whistling and watch the <boys> march.
[634] │ It was done before the <boy> could see it.
[663] │ The poor <boy> missed the boat again.
[708] │ He sent the <boy> on a short errand.Note: Parentheses in regular expressions
Quantifiers only modify the character or metacharacter immediately preceding them. So in the regular expression boys?, the ? quantifier only affects s. If we wanted this quantifier to modify y and s together, we’d need to group them with parentheses like bo(ys)?.
| means “or” in regexp-land [1] │ <a>pple
[2] │ <a>pricot
[3] │ <a>vocado
[28] │ <e>ggplant
[29] │ <e>lderberry
[53] │ <o>live
[54] │ <o>range
[79] │ <u>gli fruit[], ^ means “anything but”# Match to any fruit name starting with anything but a, e, i, o, or u
str_view(fruit, pattern = "^[^aeiou]") [4] │ <b>anana
[5] │ <b>ell pepper
[6] │ <b>ilberry
[7] │ <b>lackberry
[8] │ <b>lackcurrant
[9] │ <b>lood orange
[10] │ <b>lueberry
[11] │ <b>oysenberry
[12] │ <b>readfruit
[13] │ <c>anary melon
[14] │ <c>antaloupe
[15] │ <c>herimoya
[16] │ <c>herry
[17] │ <c>hili pepper
[18] │ <c>lementine
[19] │ <c>loudberry
[20] │ <c>oconut
[21] │ <c>ranberry
[22] │ <c>ucumber
[23] │ <c>urrant
... and 52 more[a-z], for exampleYou can use the R editor below to complete the following exercises. The editor does not save your work. If you’d prefer, you can also complete exercises in Posit Cloud.
words that contain any two vowels next to each othernumbers vector defined above, elements 1, 3, and 5 are valid 9-digit phone numbers. Write a regular expression that will match to only these strings.^.*$"\\{.+\\}""\\\\{4}"# exercise 6
str_view(words, pattern = "[aeiou]{2}")
# exercise 7
str_view(numbers, pattern = "^[0-9]{3}.[0-9]{3}.[0-9]{4}$")
# exercise 8
# part a
# start of string, zero or more characters, end of string
# part b
# the literal { character followed by any character one or more times followed by the literal } character
# part c
# the literal \ character four times in a row{stringr} cheatsheetIf you’re anything like me, you’re going to have a hard time remembering all of this regexp syntax
Luckily, there is an incredibly handy cheatsheet that you can reference. The second page contains all the useful regexp info.
I’ve been casting spells (so to speak) for awhile now and still need to consult the mystic tome (I’m enjoying this metaphor) quite frequently.
You can use the R editor below to complete the following exercises. The editor does not save your work. If you’d prefer, you can also complete exercises in Posit Cloud.
The following exercises relate to a new regular expression concept called “look-arounds”. Use the {stringr} cheatsheet (the material related to look-arounds is near the bottom of the second page) to complete these exercises. These exercises use the words vector.
str_view_all() will show multiple matches within each string.c, s, t and w characters not followed by an h using a look-around. str_view_all() will show multiple matches within each string. [1] │ a
[2] │ a<b>le
[3] │ a<b>o<u><t>
[4] │ a<b>so<l>u<t>e
[5] │ a<c>ce<p>t
[6] │ a<c>co<u><n>t
[7] │ a<c>hi<e><v>e
[8] │ a<c>ro<s>s
[9] │ a<c>t
[10] │ a<c>ti<v>e
[11] │ a<c>tu<a><l>
[12] │ a<d>d
[13] │ a<d>dre<s>s
[14] │ a<d>mi<t>
[15] │ a<d>ve<r>ti<s>e
[16] │ a<f>fe<c>t
[17] │ a<f>fo<r>d
[18] │ a<f>te<r>
[19] │ a<f>te<r>no<o><n>
[20] │ a<g>a<i><n>
... and 960 more [1] │ a
[2] │ able
[3] │ abou<t>
[4] │ ab<s>olu<t>e
[5] │ a<c><c>ep<t>
[6] │ a<c><c>oun<t>
[7] │ achieve
[8] │ a<c>ro<s><s>
[9] │ a<c><t>
[10] │ a<c><t>ive
[11] │ a<c><t>ual
[12] │ add
[13] │ addre<s><s>
[14] │ admi<t>
[15] │ adver<t>i<s>e
[16] │ affe<c><t>
[17] │ afford
[18] │ af<t>er
[19] │ af<t>ernoon
[20] │ again
... and 960 moreRegular expressions can help us transform datasets when combined with key {stringr} function.
Filtering with str_detec()
Cleaning character columns with str_replace() and str_remove()
Creating new column with str_extract()
# A tibble: 87 × 14
name height mass hair_color skin_color eye_color birth_year sex
<chr> <int> <dbl> <chr> <chr> <chr> <dbl> <chr>
1 Luke Skyw… 172 77 blond fair blue 19 male
2 C-3PO 167 75 <NA> gold yellow 112 none
3 R2-D2 96 32 <NA> white, bl… red 33 none
4 Darth Vad… 202 136 none white yellow 41.9 male
5 Leia Orga… 150 49 brown light brown 19 fema…
6 Owen Lars 178 120 brown, gr… light blue 52 male
7 Beru Whit… 165 75 brown light blue 47 fema…
8 R5-D4 97 32 <NA> white, red red NA none
9 Biggs Dar… 183 84 black light brown 24 male
10 Obi-Wan K… 182 77 auburn, w… fair blue-gray 57 male
# ℹ 77 more rows
# ℹ 6 more variables: gender <chr>, homeworld <chr>, species <chr>,
# films <list>, vehicles <list>, starships <list>Filtering
# filtering for all characters with droid names
starwars |>
filter(str_detect(name, pattern = "^[A-Z0-9]+-[A-Z0-9]+$"))# A tibble: 5 × 14
name height mass hair_color skin_color eye_color birth_year sex
<chr> <int> <dbl> <chr> <chr> <chr> <dbl> <chr>
1 C-3PO 167 75 <NA> gold yellow 112 none
2 R2-D2 96 32 <NA> white, blue red 33 none
3 R5-D4 97 32 <NA> white, red red NA none
4 IG-88 200 140 none metal red 15 none
5 R4-P17 96 NA none silver, red red, blue NA none
# ℹ 6 more variables: gender <chr>, homeworld <chr>, species <chr>,
# films <list>, vehicles <list>, starships <list>Mutating
# creating a new column that holds each films acronym
starwars |>
unnest_longer(col = films) |>
# str_extract_all() returns a vectors of extracted characters
mutate(film_acronym = str_extract_all(films,
pattern = "\\b[a-zA-z]{1}"),
# str_flatten() combines vector into single string
film_acronym = map_chr(.x = film_acronym,
.f = \(x) str_flatten(x))) |>
select(name, species, films, film_acronym)# A tibble: 173 × 4
name species films film_acronym
<chr> <chr> <chr> <chr>
1 Luke Skywalker Human A New Hope ANH
2 Luke Skywalker Human The Empire Strikes Back TESB
3 Luke Skywalker Human Return of the Jedi RotJ
4 Luke Skywalker Human Revenge of the Sith RotS
5 Luke Skywalker Human The Force Awakens TFA
6 C-3PO Droid A New Hope ANH
7 C-3PO Droid The Empire Strikes Back TESB
8 C-3PO Droid Return of the Jedi RotJ
9 C-3PO Droid The Phantom Menace TPM
10 C-3PO Droid Attack of the Clones AotC
# ℹ 163 more rowsDSC 210 Data Wrangling