Word frequency

Word frequency
You are encouraged to solve this task according to the task description, using any language you may know.

Task

Given a text file and an integer n, print the n most common words in the file (and the number of their occurrences) in decreasing frequency.

For the purposes of this task:

• A word is a sequence of one or more contiguous letters
• Uppercase letters are considered equivalent to their lowercase counterparts
• Words of equal frequency can be listed in any order

Show example output using Les Misérables from Project Gutenberg as the text file input and display the top 10 most used words.

History

This task was originally taken from programming pearls from Communications of the ACM June 1986 Volume 29 Number 6 where this problem is solved by Donald Knuth using literate programming and then critiqued by Doug McIlroy, demonstrating solving the problem in a 6 line Unix shell script.

Clojure

<lang clojure>(defn count-words [file n]

 (->> file
   slurp
   clojure.string/lower-case
   (re-seq #"\w+")
   frequencies
   (sort-by val >)
   (take n)))</lang>

Output:

user=> (count-words "135-0.txt" 10)
(["the" 41036] ["of" 19946] ["and" 14940] ["a" 14589] ["to" 13939]
 ["in" 11204] ["he" 9645] ["was" 8619] ["that" 7922] ["it" 6659])

Perl 6

Works with: Rakudo version 2017.07

This is slightly trickier than it appears initially. The task specifically states: "A word is a sequence of one or more contiguous letters", so contractions and hyphenated words are broken up. Initially we might reach for a regex matcher like /\w+/ , but \w includes underscore, which is not a letter but a punctuation connector; and this text is full of underscores since that is how Project Gutenberg texts denote italicized text. The underscores are not actually parts of the words though, they are markup.

We might try /A-Za-z/ as a matcher but this text is bursting with French words containing various accented glyphs. Those are letters, so words will be incorrectly split up; (Misérables will be counted as 'mis' and 'rables', probably not what we want.)

Actually, in this specific case /A-Za-z/ returns the correct answer since none of the French accented words or their inappropriately broken fractions are in the top 10, but that is only by accident. The correct regex matcher is some kind of Unicode aware /\w/ minus underscore.

( Really, a better regex would allow for contractions and embedded apostrophes but that is beyond the scope of this task. There are words like cat-o'-nine-tails and will-o'-the-wisps in there too to make your day even more interesting. )

<lang perl6>sub MAIN ($filename, $top = 10) {

   .say for ($filename.IO.slurp.lc ~~ m:g/[<[\w]-[_]>]+/)».Str.Bag.sort(-*.value)[^$top]

}</lang>

Output:

Passing in the file name and 10:

the => 41088
of => 19949
and => 14942
a => 14596
to => 13951
in => 11214
he => 9648
was => 8621
that => 7924
it => 6661

Python

Python2.7

<lang python>import collections import re import string import sys

def main():

 counter = collections.Counter(re.findall(r"\w+",open(sys.argv[1]).read().lower()))
 print counter.most_common(int(sys.argv[2]))

if __name__ == "__main__":

 main()</lang>

Output:

$ python wordcount.py 135-0.txt 10
[('the', 41036), ('of', 19946), ('and', 14940), ('a', 14589), ('to', 13939),
 ('in', 11204), ('he', 9645), ('was', 8619), ('that', 7922), ('it', 6659)]

Python3.6

<lang python>from collections import Counter from re import findall

les_mis_file = 'les_mis_135-0.txt'

def _count_words(fname):

   with open(fname) as f:
       text = f.read()
   words = findall(r'\w+', text.lower())
   return Counter(words)

def most_common_words_in_file(fname, n):

   counts = _count_words(fname)
   for word, count in 'WORD', 'COUNT' + counts.most_common(n):
       print(f'{word:>10} {count:>6}')

if __name__ == "__main__":

   n = int(input('How many?: '))
   most_common_words_in_file(les_mis_file, n)</lang>

Output:

How many?: 10
      WORD  COUNT
       the  41036
        of  19946
       and  14940
         a  14586
        to  13939
        in  11204
        he   9645
       was   8619
      that   7922
        it   6659

Racket

<lang racket>#lang racket

(define (all-words f (case-fold string-downcase))

 (map case-fold (regexp-match* #px"\\w+" (file->string f))))

(define (l.|l| l) (cons (car l) (length l)))

(define (counts l (>? >)) (sort (map l.|l| (group-by values l)) >? #:key cdr))

(module+ main

 (take (counts (all-words "data/les-mis.txt")) 10))</lang>

Output:

'(("the" . 41036)
  ("of" . 19946)
  ("and" . 14940)
  ("a" . 14589)
  ("to" . 13939)
  ("in" . 11204)
  ("he" . 9645)
  ("was" . 8619)
  ("that" . 7922)
  ("it" . 6659))

UNIX Shell

Works with: Bash

Works with: zsh

<lang bash>#!/bin/sh cat ${1} | tr -cs A-Za-z '\n' | tr A-Z a-z | sort | uniq -c | sort -rn | sed ${2}q</lang>

Output:

$ ./wordcount.sh 135-0.txt 10 
41089 the
19949 of
14942 and
14608 a
13951 to
11214 in
9648 he
8621 was
7924 that
6661 it