28th March 2024

A statistically improbable phrase (SIP) is a phrase or set of words that occurs more frequently in a document (or collection of documents) than in some larger corpus.[1][2][3] Amazon.com uses this concept in determining keywords for a given book or chapter, since keywords of a book or chapter are likely to appear disproportionately within that section.[4][5] Christian Rudder has also used this concept with data from online dating profiles and Twitter posts to determine the phrases most characteristic of a given race or gender in his book Dataclysm.[6] SIPs with a linguistic density of two or three words, adjective, adjective, noun or adverb, adverb, verb, will signal the author's attitude, premise or conclusions to the reader or express an important idea.

Another use of SIPs is as a detection tool for plagiarism. (Almost) unique combinations of words can be searched for online, and if they have appeared in a published text, the search will identify where. This method only checks those texts that have been published and that have been digitized online.

For example, a submission by, say, a student that contained the phrase "garden style, praising irregularity in design", might be searched for using Google.com and will yield the original Wikipedia article about Sir William Temple, English political figure and essayist.

Example

In a document about computers, the most common word is likely to be the word "the", but since "the" is the most commonly used word in the English language, it is probable that any given document will use "the" very frequently. However, a phrase like "explicit Boolean algorithm" might occur in the document at a much higher rate than its average rate in the English language. Hence, it is a phrase unlikely to occur in any given document, but did occur in the document given. "Explicit Boolean algorithm" would be a statistically improbable phrase.

Statistically improbable phrases of Darwin's On the Origin of Species could be: temperate productions, genera descended, transitional gradations, unknown progenitor, fossiliferous formations, our domestic breeds, modified offspring, doubtful forms, closely allied forms, profitable variations, enormously remote, transitional grades, very distinct species and mongrel offspring.[7]

See also

  • Collocation – Any series of words that co-occur more often than would be expected by chance
  • Googlewhack – A pair of words occurring on a single webpage, as indexed by Google
  • tf-idf – A statistic used in information retrieval and text mining
  • Complex specified information – a concept used to argue for the "intelligent design" theory

References

  1. ^ "SIPping Wikipedia" (PDF). Courses.cms.caltech.edu. Retrieved 2017-01-01.
  2. ^ Jonathan Bailey (3 July 2012). "How Long Should a Statistically Improbably Phrase Be?". Plagiarism Today.
  3. ^ Errami, Mounir; Sun, Zhaohui; George, Angela C.; Long, Tara C.; Skinner, Michael A.; Wren, Jonathan D.; Garner, Harold R. (1 June 2010). "Identifying duplicate content using statistically improbable phrases". Bioinformatics. 26 (11): 1453–1457. doi:10.1093/bioinformatics/btq146. PMC 2872002. PMID 20472545 – via bioinformatics.oxfordjournals.org.
  4. ^ "What are Statistically Improbable Phrases?". Amazon.com. Retrieved 2007-12-18.
  5. ^ Weeks, Linton (August 30, 2005). "Amazon's Vital Statistics Show How Books Stack Up". The Washington Post. Retrieved September 8, 2015.
  6. ^ Rudder, Christian (2014). Dataclysm: Who We Are When We Think No One's Looking. New York: Crown Publishers. ISBN 978-0-385-34737-2.
  7. ^ Sociologically Improbable Phrases Crooked Timber April 2005


  1. Live @ Isle Of Wight Festival, UK 15 06 2019 - Fatboy Slim 01:12:50
  2. Live @ S2O Festival, Bangkok, Thailand 13 04 2019 - Fatboy Slim 01:08:38
  3. UK TOUR 2019 - Birmingham, UK [EXTRACTS] 22 02 2019 - Fatboy Slim 42:48
  4. Brighton Rooftop B2B Set 30 11 2018 - Fatboy Eats 01:40:26
  5. Live @ Creamfields 2018, UK 26 08 2018 - Fatboy Slim 31:37
  6. Live @ Tomorrowland, Freedom Stage (Week End 2) , Belgium 28 07 2018 - Fatboy Slim 01:27:45
  7. Live @ British Airways i360, Brighton, UK 30 07 2018 - Fatboy Slim 01:29:55
  8. Live @ Tomorrowland, Belgium 21 07 2018 - Fatboy Slim 01:26:30
  9. Live @ Tuesdays On Mars, Pacha Ibiza, Spain 03 07 2018 - Fatboy Slim 03:04:03
  10. All Night Long - Pacha Ibiza 19 05 2018 - Fatboy Slim 05:16:00
  11. Record Store Day Desert Island Disco Mix, BBC 6 Music, UK 15 04 2016 - Fatboy Slim 28:31
  12. Beat Up The NME 60 Knockout Minutes (1997) Fatboy Slim 01:00:47
  13. Live @ Electric Gardens, Serafino Winery, Adelaide, Australia [EXTRACTS] 26 01 2018 - Fatboy Slim 01:04:05
  14. Live @ Hultsfred, Sweden 14 06 1997 - Fatboy Slim 01:03:14
  15. Live @ Personal Fest '17, Buenos Aires, Argentina 12 11 2017 - Fatboy Slim 01:34:54
  16. Live @ Beats for Love 2017 Festival in Ostrava, Czech Republic 05 07 2017 - Fatboy Slim 01:14:04
  17. Live @ Ultra Music Festival 2012, Miami, USA 24 03 2012 - Fatboy Slim 58:51
  18. Live @ Float Your Boat, Ibiza, Spain 21 09 2017 - Fatboy Slim 01:30:48
  19. Live @ Brixton Academy, London, UK [EXTRACTS] 20 12 2014 - Fatboy Slim 01:00:00
  20. Live @ Creamfields 2017, Liverpool, UK 27 08 2017 - Fatboy Slim 01:31:43
  21. Live @ Café Mambo, Ibiza, Spain 05 08 2017 - Fatboy Slim 38:21
  22. Live @ Tomorrowland Festival 2017,Lost Frequencies & Friends Stage,Belgium 29 07 2017 - Fatboy Slim 01:25:04
  23. Live At National Cardiff 01 01 2000 - Fatboy Slim 01:13:23
  24. Live @ Super Bock Super Rock Festival, Lisbon, Portugal 15 07 2017 - Fatboy Slim 01:30:31
  25. Live @ Kappa Futur Festival, Torino, Italy 08 07 2017 - Fatboy Slim 01:29:26
  26. Live @ Electric Daisy Carnival, Tokyo, Japan 29 04 2017 - Fatboy Slim 30:25
  27. EDC Japan 2017 Mix Fatboy Slim 01:00:00
  28. Live @ Santa Cruz Boat Party, Ibiza, Spain 15 09 2016 - Fatboy Slim 01:40:46
  29. Live @ The O² Arena, London, UK 17 12 2016 - Fatboy Slim 01:28:37
  30. Beats 1 Radio Show #8 16 12 2016 - Fatboy Slim 01:58:20
  31. Beats 1 Radio Show #7 02 12 2016 - Fatboy Slim 01:58:56
  32. Beats 1 Radio Show #6 18 11 2016 - Fatboy Slim 01:58:33
  33. Beats 1 Radio Show #5 04 11 2016 - Fatboy Slim 01:58:25
  34. Beats 1 Radio Show #4 21 10 2016 - Fatboy Slim 01:58:43
  35. Beats 1 Radio Show #3 07 10 2016 - Fatboy Slim 01:51:05
  36. Beats 1 Radio Show #2 23 09 2016 - Fatboy Slim 01:58:05
  37. Beats 1 Radio Show #1 09 09 2016 - Fatboy Slim 01:49:55
  38. DJ SET WMF, Berlin, Germany 29 09 2000 - Fatboy Slim 01:49:50
  39. BACK TO THE ODD SCHOOL feat. FATBOY SLIM's Desert Eagle Disc top ten 27 11 2016 03:00:42
  40. Essential Millennium 1999 (Disc 2) Fatboy Slim 01:16:05
  41. Live @ Boiler Room, Brighton, UK 11 10 2016 - Fatboy Slim 02:55:40
  42. One Mix: Fatboy Slim - Ep27 20 01 2016 - Beats 1 59:55
  43. Live @ Bestival 2016, Essential Mix, BBC Radio 1, UK 17 09 2016 - Fatboy Slim 01:59:48
  44. Celebrating 20 years of Better Living @ The Social, London, UK 13 09 2016 - Fatboy Slim 29:09
  45. Live At Carl Cox: Music Is Revolution, Space, Ibiza, Spain 10 08 2016 - Fatboy Slim 01:57:43
  46. Live On A Night with Annie Nightingale, Maida Vale Studio, BBC Radio 1,UK 09 09 2010 - Fatboy Slim 21:29
  47. Live @ The House Of Blues, New Orleans, USA 25 03 2000 - Fatboy Slim 02:00:03
  48. Live @ Glastonbury 2016, UK 25 06 2016 - Fatboy Slim 01:25:32
  49. From Punk Get Funk, BBC Radio 2, UK 06 06 2016 - Norman Cook 56:20
  50. Live @ The Metro, Chicago, USA 03 03 1998 – Fatboy Slim 02:03:33
  51. Live @ Club Cream 10th Birthday Special, Liverpool, UK 12 10 2002 - Fatboy Slim 41:51
  52. Live @ The Ultimate FSTVL House Party, Brighton, UK 21 04 2016 - Fatboy Slim 01:30:20
  53. Live @ HARD Haunted Mansion, Shrine Expo Hall, Los Angeles, USA 28 10 2011 - Fatboy Slim 01:11:55
  54. We Are 30 Mixed By Fatboy Slim (2013) Mixmag 01:05:29
  55. Fatboy Slim - The Legend Returns (2010) Mixmag 50:59
  56. Live At Newquay Beach, Essential Selection, BBC Radio 1, UK 21 06 2002 - Fatboy Slim & Eric Morillo 01:29:08