Natural Language Processing in R: Edit Distance

These are the notes for the second lecture in the unit on text processing. Some useful ideas like exact string matching and the definitions of characters and strings are covered in the notes of Natural Language Processing in R: Strings and Regular Expressions

Edit distance, also called Levenshtein distance, is a measure of the number of primary edits that would need to be made to transform one string into another. The R function adist() is used to find the edit distance.

adist("exactly the same","exactly the same") # edit distance 0 
adist("exactly the same","totally different") # edit distance 14

Natural Language Processing in R: Strings and Regular Expressions.

In this post, I go through a lesson in natural language processing (NLP), in R. Specifically, it covers how strings operate in R, how regular expressions work in the stringr package by Hadley Wickham, and some exercises. Included with the exercises are a list of expected hang-ups, as well as an R function that can quickly check the solutions.

This lesson is designed for a 1.5-2 hour class for senior undergrads.

Contents:

• Strings in R
• Strings can be stored and manipulated in a vector
• Strings are not factors
• Escape sequences
• The str_match() function
• Regular expressions in R
• Period . means 'any'
• Vertical line | means 'or'
• +, *, and {} define repeats
• ^ and $ mean 'beginning with' and 'ending with'
• [] is a shortcut for 'or'
• hyphens in []
• example: building a regular expression for phone numbers 

• Exercises
• Detect e-mail addresses
• Detect a/an errors
• Detect Canadian postal codes