Lab12: Text

Welcome to your LAST LAB OF THE J-TERM!

The “example” part of the lab is a little longer than usual. The good news is that it’s pretty similar to the analysis in MDSR, and should provide a lot of template code for the “on your own” portion.

We’re barely scratching the surface of text mining. If you want to learn more, I recommend Tidy Text Mining with R.

Part 1: Example

library(tidyverse)
library(tidytext)
library(gutenbergr)

lab12_theme = theme_minimal(base_family = "Optima", base_size = 12)

Today, I’m going to use novels by the Bronte sisters to walk through a few examples of basic text analysis.

Load Data

We’re going to use the gutenbergr package to download some books directly from Project Gutenberg. In order to download the right books, we need to find their IDs, which come from the URLs at their website. For instance, Jane Eyre is book 1260. We’ll store these books as `*_raw* and then clean them up later.

# 767 - Agnes Grey by Anne Bronte
# 1260 - Jane Eyre: An Autobiography by Charlotte Brontë
# 768 - Wuthering Heights by Emily Brontë

bronte_raw <- gutenberg_download(c(767, 1260, 768), mirror = "http://mirrors.xmission.com/gutenberg/", meta_fields = "title")

(You may not need the mirror argument, but the default mirror was down yesterday when I was writing this lab.)

If you don’t want to re-download the books every time you knit (you don’t), we can use an RMarkdown trick. Put the actual code for getting the books in a chunk with eval=FALSE on it and run it manually in RStudio when you want to get the data. Then you can write the downloaded data as a CSV file, and then load it invisibly from the CSV file when you knit:

I first download data from Project Gutenberg:
```{r get-book, eval=FALSE}
books_raw <- gutenberg_download(...)
write_csv(books_raw, "data/books_raw.csv")
```
```{r load-book-data-real, include=FALSE}
books_raw <- read_csv("data/books_raw.csv")
```

Clean Data

The gutenbergr package downloads data in an already tidy form, with a column for the book ID, a column for the book title (which we told it to get through meta_fields = "title"), and a column for the text. BUT, each book usually starts with a little messiness. Here, we can see that Agnes Grey starts with the title page and publisher information:

head(bronte_raw)

## # A tibble: 6 x 3
##   gutenberg_id text           title     
##          <dbl> <chr>          <chr>     
## 1          767 Agnes Grey     Agnes Grey
## 2          767 A NOVEL,       Agnes Grey
## 3          767 <NA>           Agnes Grey
## 4          767 by ACTON BELL. Agnes Grey
## 5          767 <NA>           Agnes Grey
## 6          767 LONDON:        Agnes Grey

For this lab, we’re just going to ignore this, but in the “real world”, we’d want to figure out a systematic way to remove any text that is not a part of the book. (For example, we could create a new column indicating whether each row in our dataset is part of the book or not, and then filter() based on that column)

Tokenize and count

The first step is to tokenize this data: the unnest_tokens function does this for us automatically. (The first argument is what you want the new variable to be named, the second argument is the variable we want to tokenize)

bronte_words <- bronte_raw %>% 
  drop_na(text) %>% 
  unnest_tokens(word, text)

head(bronte_words)

## # A tibble: 6 x 3
##   gutenberg_id title      word 
##          <dbl> <chr>      <chr>
## 1          767 Agnes Grey agnes
## 2          767 Agnes Grey grey 
## 3          767 Agnes Grey a    
## 4          767 Agnes Grey novel
## 5          767 Agnes Grey by   
## 6          767 Agnes Grey acton

Next, we remove stop words, count all of the words in each book and keep the top 15, and then sort them (to make the output more readable). I also include a step to make the variable an ordered factor, which is helpful for plotting.

top_words_bronte <- bronte_words %>% 
  # Remove stop words
  anti_join(stop_words) %>% 
  # Count all the words in each book
  count(title, word, sort = TRUE) %>% 
  # Keep top 15 in each book
  group_by(title) %>% 
  top_n(15) %>% 
  ungroup() %>% 
  # Make the words an ordered factor so they plot in order
  mutate(word = fct_inorder(word))
top_words_bronte

## # A tibble: 46 x 3
##    title                       word           n
##    <chr>                       <fct>      <int>
##  1 Wuthering Heights           heathcliff   421
##  2 Wuthering Heights           linton       346
##  3 Jane Eyre: An Autobiography jane         341
##  4 Wuthering Heights           catherine    336
##  5 Jane Eyre: An Autobiography rochester    317
##  6 Jane Eyre: An Autobiography sir          316
##  7 Jane Eyre: An Autobiography miss         310
##  8 Jane Eyre: An Autobiography time         244
##  9 Jane Eyre: An Autobiography day          232
## 10 Jane Eyre: An Autobiography looked       221
## # … with 36 more rows

Finally, we plot this processed data:

ggplot(top_words_bronte, aes(y = fct_rev(word), x = n, fill = title)) + 
  geom_col() + 
  guides(fill = FALSE) +
  labs(y = "Count", x = NULL, 
       title = "15 most frequent words in Bronte Novels") +
  facet_wrap(vars(title), scales = "free_y") +
  lab12_theme +
  scale_fill_viridis_d(option = "C", end = .75)

(Sharp-eyed readers will notice that the words aren’t actually in perfect order! That’s because some common words are repeated across the plays, like “miss” and “time”. However, each category in a factor can only have one possible position in the order. You can fix this with the reorder_within() function in tidytext—see Julia Silge’s tutorial here for how to use it.)

tf-idf

Now let’s try a fancy counting method (tf-idf):

bronte_words_filtered = bronte_words %>% 
  # Remove stop words
  anti_join(stop_words) %>% 
  # Count all the words in each book
  count(title, word, sort = TRUE) 

# Add the tf-idf values to the counts
bronte_tf_idf <- bronte_words_filtered %>% 
  bind_tf_idf(word, title, n)

# Get the top 10 uniquest words
bronte_tf_idf_plot <- bronte_tf_idf %>% 
  arrange(desc(tf_idf)) %>% 
  group_by(title) %>% 
  top_n(10) %>% 
  ungroup() %>% 
  mutate(word = fct_inorder(word))

ggplot(bronte_tf_idf_plot, 
       aes(y = fct_rev(word), x = tf_idf, fill = title)) +
  geom_col() +
  guides(fill = FALSE) +
  labs(x = "tf-idf", y = NULL) +
  facet_wrap(~ title, scales = "free") +
  lab12_theme +
  scale_fill_viridis_d(option = "C", end = .75)

These tf-idfs are very small compared to the example from the MDSR book on arXiv papers – this is because these three books are all much more similar to one another than the arXiv abstracts. Since the Bronte sisters are from the same era and wrote in a similar style, all tf-idf did was essentially remove non-names. Oh well!

Sentiment Analysis

A note of caution: like we saw in class, sentiment analyses can be rife with bias. They’re fun to make, and the tidytext package makes it extremely easy to do, but take results with a grain of salt.

The first step is to join the sentiment dictionary to our dataset with inner_join: this only keeps words that show up in both the dataset and the sentiment dictionary.

bronte_sentiment <- bronte_words_filtered %>% 
  inner_join(get_sentiments("bing"))
head(bronte_sentiment)

## # A tibble: 6 x 4
##   title                       word        n sentiment
##   <chr>                       <chr>   <int> <chr>    
## 1 Jane Eyre: An Autobiography miss      310 negative 
## 2 Wuthering Heights           master    185 positive 
## 3 Agnes Grey                  miss      182 negative 
## 4 Jane Eyre: An Autobiography love      151 positive 
## 5 Wuthering Heights           miss      129 negative 
## 6 Jane Eyre: An Autobiography strange    97 negative

Next, we count up each sentiment by title, and then create a new variable that is positive is sentiment is positive and negative otherwise (try running the intermediate data on your own if this doesn’t make sense). This gives as an “overall positive” and “overall negative” number for each book, which we can plot:

bronte_sentiment_by_title <- bronte_sentiment %>%
  count(title, sentiment) %>%
  mutate(
    n_pos_neg = ifelse(sentiment == "positive", n, -n)
  )

ggplot(bronte_sentiment_by_title, aes(x = sentiment, y = n_pos_neg, fill = sentiment)) +
  geom_col(position = position_dodge()) +
  facet_wrap(vars(title)) +
  lab12_theme +
  scale_fill_viridis_d(option = "C", end = .75)

We could standardize these by book length to better compare across books, but we’ll do something more interesting, which is to look at total sentiment by chapter.

First, we have to get creative in defining the “Chapter” variable - I’ve included all code to do so with comments below. The general idea is that we check if each row is equal to “CHAPTER X”, record a 1 if yes and 0 if no, and then take the cumulative sum of that new variable to get a chapter number for each line. I also did each book separately, rather than figure out a nicer way to start over at 0. (Note that this will likely be slightly different for different books!)

wuthering_heights <- bronte_raw %>% 
  filter(title == "Wuthering Heights") %>%
  # Get rid of rows where text is missing
  drop_na(text) %>% 
  # Chapters start with CHAPTER X, so mark if each row is a chapter start
  # cumsum() calculates the cumulative sum, so it'll increase every time there's
  # a new chapter and automatically make chapter numbers
  mutate(chapter_start = str_detect(text, "^CHAPTER"),
         chapter_number = cumsum(chapter_start)) %>% 
  # Get rid of these columns
  select(-chapter_start) %>%
  unnest_tokens(word, text) %>%
  anti_join(stop_words)

jane_eyre = bronte_raw %>% 
  filter(title == "Jane Eyre: An Autobiography") %>%
  # Get rid of rows where text is missing
  drop_na(text) %>% 
  # Chapters start with CHAPTER X, so mark if each row is a chapter start
  # cumsum() calculates the cumulative sum, so it'll increase every time there's
  # a new chapter and automatically make chapter numbers
  mutate(chapter_start = str_detect(text, "^CHAPTER"),
         chapter_number = cumsum(chapter_start)) %>% 
  # Get rid of these columns
  select(-chapter_start) %>%
  unnest_tokens(word, text) %>%
  anti_join(stop_words)

agnes_grey = bronte_raw %>% 
  filter(title == "Agnes Grey") %>%
  # Get rid of rows where text is missing
  drop_na(text) %>% 
  # Chapters start with CHAPTER X, so mark if each row is a chapter start
  # cumsum() calculates the cumulative sum, so it'll increase every time there's
  # a new chapter and automatically make chapter numbers
  mutate(chapter_start = str_detect(text, "^CHAPTER"),
         chapter_number = cumsum(chapter_start)) %>% 
  # Get rid of these columns
  select(-chapter_start) %>%
  unnest_tokens(word, text) %>%
  anti_join(stop_words)

bronte_with_chapters = bind_rows(wuthering_heights, jane_eyre, agnes_grey)

Next, we follow the same steps with the sentiment dictionary as before, but now counting positive and negative sentiments by book and chapter.

bronte_sentiment_by_chapter <- bronte_with_chapters %>%
  inner_join(get_sentiments("bing")) %>%
  group_by(title, chapter_number) %>%
  summarize(
    n_pos = sum(sentiment == "positive"),
    n_neg = sum(sentiment == "negative")
  ) %>%
  mutate(
    sent_diff = n_pos - n_neg
  )

ggplot(bronte_sentiment_by_chapter, aes(x = chapter_number, y = sent_diff, fill = sent_diff > 0)) +
  geom_col() + 
  facet_wrap(vars(title)) +
  lab12_theme +
  scale_fill_viridis_d(option = "C", end = .75) +
  guides(fill = FALSE)

Now this is more interesting! We can see that Wuthering Heights is a real bummer throughout, but both Agnes Grey and Jane Eyre end on a positive note.

Bigram Analysis

The last example I’ll walk through is a bigram analysis. An n-gram is a set of n words, and so a bigram is a set of 2 words. Here’s how to use the unnest_tokens function to count bigrams instead of words:

bronte_bigrams <- bronte_raw %>% 
  drop_na(text) %>% 
  # n = 2 here means bigrams. We could also make trigrams (n = 3) or any type of n-gram
  unnest_tokens(bigram, text, token = "ngrams", n = 2) %>% 
  # Split the bigrams into two words so we can remove stopwords
  separate(bigram, c("word1", "word2"), sep = " ") %>% 
  filter(!word1 %in% stop_words$word,
         !word2 %in% stop_words$word) %>% 
  # Put the two word columns back together
  unite(bigram, word1, word2, sep = " ")
bronte_bigrams

## # A tibble: 29,103 x 3
##    gutenberg_id title      bigram         
##           <dbl> <chr>      <chr>          
##  1          767 Agnes Grey agnes grey     
##  2          767 Agnes Grey acton bell     
##  3          767 Agnes Grey NA NA          
##  4          767 Agnes Grey thomas cautley 
##  5          767 Agnes Grey cautley newby  
##  6          767 Agnes Grey newby publisher
##  7          767 Agnes Grey 72 mortimer    
##  8          767 Agnes Grey mortimer st    
##  9          767 Agnes Grey st cavendish   
## 10          767 Agnes Grey cavendish sq   
## # … with 29,093 more rows

Next, we follow the same counting steps as before, but adding the extra step of taking out “NA NA” since that appears in our data.

top_bigrams <- bronte_bigrams %>% 
  filter(bigram != "NA NA") %>%
  # Count all the bigrams in each play
  count(title, bigram, sort = TRUE) %>% 
  # Keep top 15 in each play
  group_by(title) %>% 
  top_n(15) %>% 
  ungroup() %>% 
  # Make the bigrams an ordered factor so they plot in order
  mutate(bigram = fct_inorder(bigram))

ggplot(top_bigrams, aes(y = fct_rev(bigram), x = n, fill = title)) + 
  geom_col() + 
  guides(fill = FALSE) +
  labs(y = "Count", x = NULL, 
       title = "15 most frequent bigrams in Bronte novels") +
  facet_wrap(vars(title), scales = "free") + 
  lab12_theme + 
  scale_fill_viridis_d(option = "C", end = .75) +
  guides(fill = FALSE)

Now we can see where all of the “miss”es came from! (I will also point out couldn't bear and i'm tired are top bigrams in Wuthering Heights, which is more evidence to the overall bummer-ness of the book)

Next, let’s try a version of She giggles, he gallops, by restricting the dataset to only include bigrams that start with “he” or “she”:

pronoun_bigrams <- bronte_raw %>% 
  drop_na(text) %>% 
  unnest_tokens(bigram, text, token = "ngrams", n = 2) %>% 
  # Split the bigrams into two words so we can remove stopwords
  separate(bigram, c("word1", "word2"), sep = " ") %>% 
  # only filtering word 2, since we want "he" and "she" to be able to show up in first word
  filter(!word2 %in% stop_words$word) %>% 
  # Put the two word columns back together
  unite(bigram, word1, word2, sep = " ") %>%
  filter(bigram != "NA NA") %>%
  # Find bigrams that start with he or she and end with a space
  filter(str_detect(bigram, "^he ") | str_detect(bigram, "^she ")) %>%
  # Count all the bigrams in each play
  count(title, bigram, sort = TRUE) %>% 
  # Keep top 15 in each play
  group_by(title) %>% 
  top_n(15) %>% 
  ungroup() %>% 
  # Make the bigrams an ordered factor so they plot in order
  mutate(bigram = fct_inorder(bigram))

pronoun_verbs = pronoun_bigrams %>% 
  separate(bigram, into = c("pronoun", "verb")) %>%
  mutate(
    n_pos_neg = ifelse(pronoun == "he", n, -n)
  ) %>%
  group_by(title, verb) %>%
  summarize(n_male = sum(n_pos_neg))

ggplot(pronoun_verbs, aes(y = fct_rev(verb), x = n_male, fill = n_male > 0)) + 
  geom_col() + 
  guides(fill = FALSE) +
  labs(y = "Count", x = NULL, 
       title = "Most Common Verbs used with he vs she pronoun",
       subtitle = "Positive = More associated with he") +
  facet_wrap(vars(title), scales = "free") + 
  lab12_theme + 
  scale_fill_viridis_d(option = "C", end = .75) +
  guides(fill = FALSE)

This isn’t super interesting in this case, since most characters just talk to one another in these books, but maybe you will pick books with more interesting verbs!

Part 2: On your own

Download 3+ books by some author on Project Gutenberg. Jane Austen, Victor Hugo, Lucy Maud Montgomery, Arthur Conan Doyle, Mark Twain, Henry David Thoreau, Fyodor Dostoyevsky, Leo Tolstoy. Anyone. Just make sure it’s all from the same author. If you are able to find downloadable books elsewhere (like the Harry Potter example from class) and prefer to do that, that’s OK too! You just might have to do a bit more start up work to get the data into a tidy format.

Make these two plots and describe what each tell about your author’s books:

1. Top 10 most frequent words in each book OR top 10 most unique words in each book (i.e. tf-idf)
2. Any other plot of your choice

You can knit your .Rmd file to any file format you wish!