Getting started

What is the RAKE algorithm?

The Rapid Automatic Keyword Extraction (RAKE) algorithm was first described in Rose et al. as a way to quickly extract keywords from documents. The algorithm invovles two main steps:

1. Identify candidate keywords. A candidate keyword is any set of contiguous words (i.e., any n-gram) that doesn’t contain a phrase delimiter or stop word.1 A phrase delimiter is a punctuation character that marks the beginning or end of a phrase (e.g., a period or comma). Splitting up text based on phrase delimiters/stop words is the essential idea behind RAKE. According to the authors:

RAKE is based on our observation that keywords frequently contain multiple words but rarely contain standard punctuation or stop words, such as the function words and, the, and of, or other words with minimal lexical meaning

In addition to using stop words and phrase delimiters to identify the candidate keywords, you can also use a word’s part-of-speech (POS). For example, most keywords don’t contain verbs, so you may want treat verbs as if they were stop words. You can use slowrake()’s stop_pos parameter to choose which parts-of-speech to exclude from your candidate keywords.

2. Calculate each keyword’s score. A keyword’s score (i.e., its degree of “keywordness”) is the sum of its member word scores. For example, the score for the keyword “dog leash” is calculated by adding the score for the word “dog” with the score for the word “leash.” A member word’s score is equal to its degree/frequency, where degree equals the number of times that the word co-occurs with other words (in other keywords), and frequency is the total number of times that the word occurs overall.

See Rose et al. for more details on how RAKE works.

Examples

RAKE is unique in that it is completely unsupervised (i.e., no training data is required), so it’s relatively easy to use. Let’s take a look at a few basic examples that demonstrate slowrake()’s various parameters.

library(slowraker)

txt <- "Compatibility of systems of linear constraints over the set of natural numbers. Criteria of compatibility of a system of linear Diophantine equations, strict inequations, and nonstrict inequations are considered. Upper bounds for components of a minimal set of solutions and algorithms of construction of minimal generating sets of solutions for all types of systems are given. These criteria and the corresponding algorithms for constructing a minimal supporting set of solutions can be used in solving all the considered types of systems and systems of mixed types."
slowrake(txt = txt)[[1]]
#>                         keyword freq score                    stem
#> 1  linear diophantine equations    1   8.5 linear diophantin equat
#> 2        minimal supporting set    1   6.8       minim support set
#> 3            linear constraints    1   4.5       linear constraint
#> 4               natural numbers    1   4.0            natur number
#> 5         nonstrict inequations    1   4.0         nonstrict inequ
#> 6            strict inequations    1   4.0            strict inequ
#> 7                   minimal set    1   3.8               minim set
#> 8                   mixed types    1   3.3                mix type
#> 9                       minimal    1   2.0                   minim
#> 10                          set    1   1.8                     set
#> 11                         sets    1   1.8                     set
#> 12                        types    2   1.3                    type
#> 13                   algorithms    2   1.0               algorithm
#> 14                compatibility    2   1.0                  compat
#> 15                   components    1   1.0                  compon
#> 16                 construction    1   1.0               construct
#> 17                     criteria    2   1.0                criteria
#> 18                    solutions    3   1.0                   solut
#> 19                       system    1   1.0                  system
#> 20                      systems    4   1.0                  system
#> 21                        upper    1   1.0                   upper
slowrake(txt = txt, stem = FALSE)[[1]]
#>                         keyword freq score
#> 1  linear diophantine equations    1   8.5
#> 2        minimal supporting set    1   7.0
#> 3            linear constraints    1   4.5
#> 4                   minimal set    1   4.0
#> 5               natural numbers    1   4.0
#> 6         nonstrict inequations    1   4.0
#> 7            strict inequations    1   4.0
#> 8                   mixed types    1   3.3
#> 9                       minimal    1   2.0
#> 10                          set    1   2.0
#> 11                        types    2   1.3
#> 12                   algorithms    2   1.0
#> 13                compatibility    2   1.0
#> 14                   components    1   1.0
#> 15                 construction    1   1.0
#> 16                     criteria    2   1.0
#> 17                         sets    1   1.0
#> 18                    solutions    3   1.0
#> 19                       system    1   1.0
#> 20                      systems    4   1.0
#> 21                        upper    1   1.0
slowrake(txt = txt, stop_words = c(smart_words, "diophantine"))[[1]]
#>                   keyword freq score              stem
#> 1  minimal supporting set    1   6.8 minim support set
#> 2         natural numbers    1   4.0      natur number
#> 3   nonstrict inequations    1   4.0   nonstrict inequ
#> 4      strict inequations    1   4.0      strict inequ
#> 5             minimal set    1   3.8         minim set
#> 6      linear constraints    1   3.5 linear constraint
#> 7             mixed types    1   3.3          mix type
#> 8                 minimal    1   2.0             minim
#> 9                     set    1   1.8               set
#> 10                   sets    1   1.8               set
#> 11                 linear    1   1.5            linear
#> 12                  types    2   1.3              type
#> 13             algorithms    2   1.0         algorithm
#> 14          compatibility    2   1.0            compat
#> 15             components    1   1.0            compon
#> 16           construction    1   1.0         construct
#> 17               criteria    2   1.0          criteria
#> 18              equations    1   1.0             equat
#> 19              solutions    3   1.0             solut
#> 20                 system    1   1.0            system
#> 21                systems    4   1.0            system
#> 22                  upper    1   1.0             upper
slowrake(txt = txt, stop_pos = NULL)[[1]]
#>                         keyword freq score                    stem
#> 1  linear diophantine equations    1   8.5 linear diophantin equat
#> 2       minimal generating sets    1   7.9         minim gener set
#> 3        minimal supporting set    1   7.9       minim support set
#> 4                   minimal set    1   4.9               minim set
#> 5            linear constraints    1   4.5       linear constraint
#> 6               natural numbers    1   4.0            natur number
#> 7         nonstrict inequations    1   4.0         nonstrict inequ
#> 8            strict inequations    1   4.0            strict inequ
#> 9                  upper bounds    1   4.0             upper bound
#> 10                  mixed types    1   3.7                mix type
#> 11             considered types    1   3.2             consid type
#> 12                          set    1   2.2                     set
#> 13                        types    1   1.7                    type
#> 14                   considered    1   1.5                  consid
#> 15                   algorithms    2   1.0               algorithm
#> 16                compatibility    2   1.0                  compat
#> 17                   components    1   1.0                  compon
#> 18                 constructing    1   1.0               construct
#> 19                 construction    1   1.0               construct
#> 20                     criteria    2   1.0                criteria
#> 21                    solutions    3   1.0                   solut
#> 22                      solving    1   1.0                    solv
#> 23                       system    1   1.0                  system
#> 24                      systems    4   1.0                  system
slowrake(txt = txt, stop_pos = pos_tags$tag[!grepl("^N", pos_tags$tag)])[[1]]
#>          keyword freq score       stem
#> 1     algorithms    2     1  algorithm
#> 2  compatibility    2     1     compat
#> 3     components    1     1     compon
#> 4    constraints    1     1 constraint
#> 5   construction    1     1  construct
#> 6       criteria    2     1   criteria
#> 7      equations    1     1      equat
#> 8    inequations    2     1      inequ
#> 9        numbers    1     1     number
#> 10           set    3     1        set
#> 11          sets    1     1        set
#> 12     solutions    3     1      solut
#> 13        system    1     1     system
#> 14       systems    4     1     system
#> 15         types    3     1       type
#> 16         upper    1     1      upper
res <- slowrake(txt = txt)[[1]]
res2 <- aggregate(freq ~ keyword + stem, data = res, FUN = sum)
res2[order(res2$freq, decreasing = TRUE), ]
#>                         keyword                    stem freq
#> 19                      systems                  system    4
#> 16                    solutions                   solut    3
#> 1                    algorithms               algorithm    2
#> 2                 compatibility                  compat    2
#> 5                      criteria                criteria    2
#> 20                        types                    type    2
#> 3                    components                  compon    1
#> 4                  construction               construct    1
#> 6            linear constraints       linear constraint    1
#> 7  linear diophantine equations linear diophantin equat    1
#> 8                       minimal                   minim    1
#> 9                   minimal set               minim set    1
#> 10       minimal supporting set       minim support set    1
#> 11                  mixed types                mix type    1
#> 12              natural numbers            natur number    1
#> 13        nonstrict inequations         nonstrict inequ    1
#> 14                          set                     set    1
#> 15                         sets                     set    1
#> 17           strict inequations            strict inequ    1
#> 18                       system                  system    1
#> 21                        upper                   upper    1
slowrake(txt = dog_pubs$abstract[1:10])
#> 
  |                                                                       
  |                                                                 |   0%
  |                                                                       
  |======                                                           |  10%
  |                                                                       
  |=============                                                    |  20%
  |                                                                       
  |====================                                             |  30%
  |                                                                       
  |==========================                                       |  40%
  |                                                                       
  |================================                                 |  50%
  |                                                                       
  |=======================================                          |  60%
  |                                                                       
  |==============================================                   |  70%
  |                                                                       
  |====================================================             |  80%
  |                                                                       
  |==========================================================       |  90%
  |                                                                       
  |=================================================================| 100%
#> 
#> # A rakelist containing 10 data frames:
#>  $ :'data.frame':    61 obs. of  4 variables:
#>   ..$ keyword:"assistance dog identification tags" ...
#>   ..$ freq   :1 1 ...
#>   ..$ score  :11 ...
#>   ..$ stem   :"assist dog identif tag" ...
#>  $ :'data.frame':    90 obs. of  4 variables:
#>   ..$ keyword:"current dog suitability assessments focus" ...
#>   ..$ freq   :1 1 ...
#>   ..$ score  :21 ...
#>   ..$ stem   :"current dog suitabl assess focu" ...
#> #...With 8 more data frames.

  1. Technically, the original version of RAKE allows some keywords to contain stop words, but slowrake() does not allow for this.