In this article, I will show you how to use text data to build word clouds in R. We will use a dataset containing around 200k Jeopardy questions. The dataset can be downloaded here (thanks to reddit user trexmatt for providing the dataset).
We will require three packages for this: tm, SnowballC, and wordcloud.
First, let’s load the required libraries and read in the data.
library(tm)
library(SnowballC)
library(wordcloud)
jeopQ <- read.csv('JEOPARDY_CSV.csv', stringsAsFactors = FALSE)
The actual questions are available in the Question column.
Now, we will perform a series of operations on the text data to simplify it.
First, we need to create a corpus.
jeopCorpus <- Corpus(VectorSource(jeopQ$Question))
Next, we will convert the corpus to a lowercase.
jeopCorpus <- tm_map(jeopCorpus, content_transformer(tolower))
Then, we will remove all punctuation and stopwords, and convert it to a plain text document.. Stopwords are commonly used words in the English language such as I, me, my, etc. You can see the full list of stopwords using stopwords('english').
jeopCorpus <- tm_map(jeopCorpus, removePunctuation)
jeopCorpus <- tm_map(jeopCorpus, PlainTextDocument)
jeopCorpus <- tm_map(jeopCorpus, removeWords, stopwords('english'))
Next, we will perform stemming. This means that all the words are converted to their stem (Ex: learning -> learn, walked -> walk, etc.). This will ensure that different forms of the word are converted to the same form and plotted only once in the wordcloud.
jeopCorpus <- tm_map(jeopCorpus, stemDocument)
Now, we will plot the wordcloud.
wordcloud(jeopCorpus, max.words = 100, random.order = FALSE)
This will produce the following wordcloud:
There are a few ways to customize it.
- scale: This is used to indicate the range of sizes of the words.
- max.words and min.freq: These parameters are used to limit the number of words plotted. max.words will plot the specified number of words and discard least frequent terms, whereas, min.freq will discard all terms whose frequency is below the specified value.
- random.order: By setting this to FALSE, we make it so that the words with the highest frequency are plotted first. If we don’t set this, it will plot the words in a random order, and the highest frequency words may not necessarily appear in the center.
- rot.per: This value determines the fraction of words that are plotted vertically.
- colors: The default value is black. If you want to use different colors based on frequency, you can specify a vector of colors, or use one of the pre-defined color palettes. You can find a list here.
That brings us to the end of this article. I hope you enjoyed it! As always, if you have questions, feel free to leave a comment or reach out to me on Twitter.
Edit: After struggling a lot, I resorted to StackOverflow for a fix. I forgot to convert the document to lower case. As explained in the StackOverflow thread, a lot of the words start with “The”, with an uppercase “T”, whereas stopwords has “the” with a lowercase “t”. This is what was causing the words “the” and “this” to appear in the wordcloud.
Note: I learnt this technique in The Analytics Edge course offered by MIT on edX. It is a great course and I highly recommend that you take it if you are interested in Data Science!
textCloud <- function(text, lang = "english", exclude = c(), seed = 0, print = T, keep_spaces = FALSE) {
require("tm")
require("SnowballC")
require("wordcloud")
require("RColorBrewer")
set.seed(seed)
options(warn=-1)
text <- as.character(text)
if (keep_spaces == TRUE) {
text <- gsub(" ", "_", text) # '_' deleted later on
}
## Load the data as a corpus
docs <- Corpus(VectorSource(text))
## Text transformation
toSpace <- content_transformer(function (x , pattern) gsub(pattern, " ", x))
docs <- tm_map(docs, toSpace, "/")
docs <- tm_map(docs, toSpace, "@")
docs <- tm_map(docs, toSpace, "\|")
## Cleaning the text
# Convert the text to lower case
docs <- tm_map(docs, content_transformer(tolower))
# Remove numbers
docs <- tm_map(docs, removeNumbers)
# Remove english common stopwords
docs <- tm_map(docs, removeWords, stopwords(lang))
# Remove your own stop word (specify your stopwords as a character vector)
docs <- tm_map(docs, removeWords, rbind("https", exclude))
# Remove punctuations
docs <- tm_map(docs, removePunctuation)
# Eliminate extra white spaces
docs <- tm_map(docs, stripWhitespace)
## Build a term-document matrix
dtm <- TermDocumentMatrix(docs)
m <- as.matrix(dtm)
v <- sort(rowSums(m), decreasing=TRUE)
d <- data.frame(word = names(v), freq=v)
if (print == TRUE) {
message(paste0(capture.output(head(d, 10)), collapse = "n"))
}
wordcloud(words = d$word, freq = d$freq,
scale = c(3.5, .7),
min.freq = 1,
max.words = 200,
random.order = FALSE,
rot.per = 0.2,
colors = brewer.pal(8, "Paired"))
}
Hope you find this useful 😉
Hi, I am unable to set a title to the worcloud.For example, I want to set the title as “Word Topic probabilities” and plot a wordcloud followed. How do I do this?
Not sure if this will still be checked, but I am using RStudio Version 1.0.153 is there a way to do this on that? I can’t figure out how to download R 3.4.1
R version 3.4.0 (2017-04-21)
Got two warnings:
package ‘tm’ was built under R version 3.4.1
package ‘wordcloud’ was built under R version 3.4.1
No other warnings while running the code as given by you. When it comes to the last line, wordcloud(jeopCorpus, max.words = 100, random.order = FALSE)
Error in simple_triplet_matrix(i, j, v, nrow = length(terms), ncol = length(corpus), :
‘i, j’ invalid
What’s with these words in the cloud:
countri, citi, includ, mani, lill, hous, becam, charact, compani, titl, centuri,featur, etc.
Also- why is “the” showing up in the first place? Isn’t it one of the contained stopwords in stopwords(‘english’)?
I was just informed that I forgot to convert it to lower case. A lot of the questions start with “The”, with an uppercase T, and “the” in stopwords has lowercase t, which is why it wasn’t removed. I have updated the code to reflect this.
Hey Teja,
Is there a way to specify which column you want to perform the word cloud on? For example, if you only wanted to create a wordcloud on the “Answer” column and not the whole dataframe?
Thanks.
Hello Ryan, if you look at the first line of code where we create the corpus, you can see that we created it using the question column. I’m afraid that the only way to do it with the answer column is just to use jeopQ$Answer instead of jeopQ$Question and repeat the rest of the code.
Nice post…I ran into what seems to be a problem reported by others too though. Running tm under R 3.2.2 the removeWords step results in…
jeopCorpus inspect(jeepCorpus[11:15])
Error in inspect(jeepCorpus[11:15]) : object ‘jeepCorpus’ not found
> inspect(jeopCorpus[11:15])
The preceding steps executed correctly (I used inspect to look at the output).
> jeopQ jeopCorpus jeopCorpus jeopCorpus <- tm_map(jeopCorpus, PlainTextDocument)
Any thoughts on how this could be fixed? Thanks GC
It’s really interesting. The various OS and R updates I ran since I posted this question seem to have taken care of the problem. Teja’s suggestion was to try adding lazy=true to the tm_map statements but that wasn’t necessary in the end. Thanks!!!
Nice. But no matter what I do, I cannot get rid of the words “this” and “the”. I have gone so far as to quit and restart R (so I would have nothing extra loaded in my environment) as well as to break your: line: jeopCorpus <- tm_map(jeopCorpus, removeWords, c('the', 'this', stopwords('english')))
into two separate commands:
jeopCorpus <- tm_map(jeopCorpus, removeWords, stopwords('english'))
jeopCorpus <- tm_map(jeopCorpus, removeWords, c('the', 'this'))
but when I plot the wordcloud, I still see "the" and "this"
?? Thanks!
Hello Wayne, thank you for bringing this to my attention. It seems that running the commands in the wrong order is causing this to happen. I just ran the Corpus and the removeWords command (with only the word ‘the’) and I managed to make the word ‘the’ go away. I need to do some more tests, and I will update the article with a more concrete solution tomorrow.
I’m confused. Why is “the” showing up in the first place? Isn’t it one of the contained stopwords in stopwords(‘english’)?
Nice article
Did you learn how to use the term_score function in the course? Would you recommend it?
Hello Syed, I am unaware of that function, sorry about that.