Posted: September 29th, 2020 | Author: Domingo | Filed under: Artificial Intelligence | Tags: AI, Everett, NLP, NLU, Zipf | Comments Off on Language is a Rum Thing
Zipf and His Word Frequency Distribution Law
Although it might sound surprising for some data scientists, the supposedly successful use of machine learning techniques to tackle the problem of natural language processing is based on the work of an US philologist called George Kingsley Zipf (1902-1950). Zipf analyzed the frequency distribution of certain terms and words in several languages, enunciating the law named after him in the 40’s of the past century. Ah, these crazy linguists!
One of the most puzzling facts about human language is also one of the most basic: Words occur according to a famously systematic frequency distribution such that there are few very high-frequency words that account for most of the tokens in text (e.g., “a,” “the,” “I,” etc.) and many low-frequency words (e.g., “accordion,” “catamaran,” “jeopardize”). What is striking is that the distribution is mathematically simple, roughly obeying a power law known as Zipf’s law: The rth most frequent word has a frequency f(r) that scales according to
f(r)∝1/rα
for α≈1 (Zipf, 1932, 1936)(1) In this equation, r is called the frequency rank of a word, and f(r) is its frequency in a natural corpus. Since the actual observed frequency will depend on the size of the corpus examined, this law states frequencies proportionally: The most frequent word (r = 1) has a frequency proportional to 1, the second most frequent word (r = 2) has a frequency proportional to 1/2α, the third most frequent word has a frequency proportional to 1/3α, and so forth.
From Zipf`s standpoint as well, the length of a word, far from being a random matter, is closely related to the frequency of its usage -the greater the frequency, the shorter the word. The more complex any speech-element is phonetically, the less frequent it occurs. In English the most frequent word in the sample will occur on the average once in approximately every 10 words; the second most frequent word once in every 20 words; the third most frequent word once in every 1,000 words; in brief, the distribution of words in English approximates with remarkable precision an harmonic series. Similarly, one finds in English (or Latin or Chinese) the following striking correlation. If the number of different words occurring once in a given sample is taken as x, the number of different words occurring twice, three times, four times, n times, in the same sample, is respectively 1/22, 1/32, 1/42… 1/n2 of x, up to, though not including, the few most frequently used words; that is, an unmistakable progression according to the inverse square is found, valid for over 95% of all the different words used in the sample.
This evidence points to the existence of a fundamental condition of equilibrium between the form and function of speech-habits, or speech-patterns, in any language. The impulse to preserve or restore this condition of equilibrium is the underlying cause of linguistic change. All speech-elements or language-patterns are impelled and directed in their behavior by a fundamental law of economy, in which there is the desire to maintain an equilibrium between form and behavior, always according to Zipf.
Nonetheless, if our languages are pure statistical distributions, what happens with meanings? Is there a multiplicative stochastic process at play? Absolutely not! We select and arrange our words according to their meanings with little or no conscious reference to the relative frequency of occurrence of those words in the stream of speech, yet we find that words thus selected and arranged have a frequency distribution of great orderliness which for a large portion of the curve seems to be constant for language in general. The question arises as to the nature of the meaning or meanings which leads automatically to this orderly frequency distribution.
A study of language is certainly incomplete which totally disregards all questions of meaning, emotion, and culture even though these refer to the most elusive of mental phenomena.
Daniel Everett and Language as a Cultural Tool
According to the linguist Everett, language is an artifact, a cultural tool, an instrument created by hominids to satisfy their social need of meaning and community (Everett, 2013)(2).
Linguists, psychologists, anthropologists, biologists, and philosophers tend to divide into those who believe that human biology is endowed with a language-dedicated genetic program and those who believe instead that human biology and the nature of the world provide general mechanisms, that allow us the flexibility to acquire a large array of general skills and abilities of which language is but one. The former often refers to a “language instinct” or a “universal grammar” (Chomsky dixit) shared by all humans. The latter talk about learning language as we learn many other skills, such as cooking, chess, or carpentry. The latter proposal takes seriously the idea that the function of language shapes its form. It recognizes the linguistic importance of the utilitarian forces radiating from the human necessity to communicate in order to survive. Language emerges as the nexus of our biological endowment and our environmental existence.
According to Chomsky meaning is secondary to grammar and all we need to understand of a formal grammar is that if we follow the rules and combine the symbols properly, then the sentences generated are grammatical -does it sound familiar to the ML approach to NLP?. Nonetheless, this is not accurate: beings with just a grammar would not have language. In fact, we know that meaning drives most, if not all the grammar. Meaning would have to appear at least as early in the evolution of language as grammar.
Forms in language vary radically and thus serve to remind us that humans are the only species with a communication system whose main characteristics is variation and not homogeneity. Humans do not merely produce fixed calls like vervet monkeys, they fit their messages to specific contexts and intentions.
People organize their words by related meanings -semantic fields-, by sound structure, by most common meanings, and so on. Even our verb structures are constrained by our cultures and what these cultures consider to be an “effable event”. For instance, the Pirahãs -an indigenous people of the Amazon Rainforest in Brazil- do not talk about the distant past or the far-off future because a cultural value of theirs is to talk only about the present or the short-term past or future.
Can grammatical structure itself be shaped by culture? Let’s consider another example: researchers claim there is no verb “to give” in Amele mainly for cultural reasons: giving is so basic to Amele culture the language manifests a tendency to allow the “experiential basicness” of giving to correspond to a “more basic kind of linguistic form” – that is zero. No verb is needed for this fundamental concept of Amele culture.
Language has been shaped in its very foundation by our socio-cultural needs. Languages fit their cultural niches and take on the properties required of them in their environments. That is one reason that languages change over time -they evolve to fit new cultural circumstances.
Our language is shaped to facilitate communication. There is very little evidence for arbitrariness in the design of grammars. People both overinterpret and under-interpret what they hear based on cultural expectations built into their communication patterns. We learn to predict, by means of what some researchers think is a sophisticated and unconscious computational computation of probabilities what a speaker is likely to say next once we learn that the relationships amongst words are contingent what the likehood of one word following another one is. Crucial for language acquisition is what we call the “interactional instinct”. This instinct is at innate drive amongst human infants to interact with conspecific caregivers. Babies and children learn from their parents’ faces what is in their parents’ minds and they adjust their own inner mental lives accordingly. Rather than learning algebraic procedures for combining symbols, children instead seem to learn linguistic categories and constructions as patterns of meaningful symbols.
All humans belong to culture and share values and knowledge with other members of their cultures. With the current approach an AI/NLP model will never be able to learn culture. Therefore, it can never learn a language stricto sensu, though it can learn lists of grammatical rules and lexical combinations.
Without culture, no background, without background no signs, without signs, no stories and no language.
Recapping, it seems NLP keeps on being the last challenge for AI practitioners and aficionados. Blending the mathematical-statistical and tbe symbolic approaches is paramount to find a solution to this conundrum. I’m positive the moment we succeed, we’ll be closer to strong AI… Still a long way ahead.
Die Grenzen Meiner Sprache sind die Grenzen meiner Welt. Ludwig Wittgenstein (1889 – 1951).
Bibliography:
(1) The Psycho-Biology of Language. An Introduction to Dynamic Philology. George Kingsley Zipf. 1936
Selected Studies of the Principle of Relative Frequency in Language. George Kingsley Zipf. 1932,
(2) Language. The Cultural Tool. Daniel Everett, 2013. Profile Books.
