Formal grammar and information theory
Natural Language is a complex entity and in order to process it through a computer based program, we need to build a model of it. Two popular approaches of building a language model are:-
- Rationalist: It is characterised by the belief that a significant part of the knowledge in the human mind is not derived by the senses but is fixed in advance.
- Empiricist: This approach assumes that a baby’s brain begins with general operations for association, pattern recognition, and generalisation, and that these can be applied to the rich sensory input available to the child to learn the detailed structure of natural language.
Within artificial intelligence, rationalist beliefs can be seen as supporting the attempt to create intelligent systems by inserting starting knowledge and reasoning mechanisms e.g. rules in grammar-based modeling. An empiricist approach also begins by postulating some cognitive abilities as present in the brain but focuses more on estimating the probability distribution as accurately as possible e.g. statistical language modeling techniques using machine learning.
The machine learning paradigm calls for using statistical inference to automatically learn rules of language through the analysis of large corpora of typical real-world examples. ML models have a representational capacity which can…
