Chomsky Hierarchy Explained | Computerphile

This lecture covers two fundamental parsing techniques in compiler design: (1) LL1 parsing, a top-down approach using deterministic push-down automata where parsing tables are constructed using First Set and Follow Set calculations to resolve production selection, with LL1 grammars requiring single entries per table cell; (2) LR0 item DFA construction for bottom-up parsing, which uses Closure and Goto functions to build deterministic finite automata for shift-reduce parsing, where Closure adds productions for non-terminals following the dot and Goto moves the dot one position to the right.

LL(1) grammars must be context-free, unambiguous, left-recursion-free, and left-factored, but satisfying all four properties does not guarantee LL(1). Detection shortcuts: if First(α1) ∩ First(α2) ≠ ∅ for non-epsilon productions, or First(α) ∩ Follow(A) ≠ ∅ for epsilon productions, the grammar is not LL(1). Left factoring eliminates common prefixes to enable LL(1) parsing. LR grammars are classified as LR(0), SLR(1), LALR(1), and CLR(1), where L stands for left-to-right scan and R for rightmost derivation in reverse. The hierarchy is: LR(0) ⊂ SLR(1) ⊂ LALR(1) ⊂ CLR(1). LL(1) grammars (without null productions) are a subset of SLR(1). Conflicts: shift-reduce conflicts occur when a state has both shift and reduce items; reduce-reduce conflicts occur when a state has multiple reduce items.

Context-free language parsing includes LL (top-down) and LR (bottom-up) families. Both use table-driven approaches with parsing tables and stacks. The algorithm is static, with complexity in building the table. LL parsers use first and follow sets to determine which production to apply. LR parsers are more expressive but larger. Different variants (SLR, LALR, LR) have different expressiveness levels. The speaker emphasizes that the algorithm is simple but data structure building is challenging.

Syntax analysis (parsing) is the phase in compiler design that verifies whether tokens generated by the lexical analyzer are grouped according to the syntax rules of the language, using a Context-Free Grammar (CFG) to construct a parse tree for a given input string; parsers are classified into two main types: top-down parsers (including backtracking and predictive parsers like LL parsers) and bottom-up parsers (including operator precedence parsers, shift-reduce parsers, and LR parsers like LR(0), SLR, LALR, and Canonical LR).

Syntax analysis is the second compiler phase, following lexical analysis. It takes tokens and Context-Free Grammar (CFG) as input, validates tokens against the grammar, and generates a parse tree or syntax error. A parser performs this validation. Two main approaches exist: Top-Down (root to leaves) and Bottom-Up (leaves to root). Top-Down includes Brute Force (with backtracking, exponential complexity) and LL1 (deterministic, no backtracking). Bottom-Up includes LR Parsing (LR0, SLR1, CLR1, LR1) and Operator Precedence. LL1 notation: First 'L' = Left-to-Right scanning, Second 'L' = Leftmost derivation, '1' = One look-ahead symbol. Components: Input Buffer, Stack, Dollar Symbol ($), Parser Program, and LL1 Parsing Table. Working follows three steps: (1) If top symbol equals look-ahead AND look-ahead is $, input is valid; (2) If top symbol equals look-ahead but not $, increment look-ahead; (3) If top is non-terminal, consult LL1 Parsing Table for production. If table entry exists, replace non-terminal with production RHS; if no entry, syntax error.

P is the class of decision problems solvable by deterministic Turing machines in polynomial time (efficiently computable). NP is the class where proposed solutions can be verified efficiently, or equivalently, solvable by non-deterministic Turing machines in polynomial time. The fundamental question P vs NP asks whether every efficiently verifiable problem is also efficiently solvable. NP-complete problems are the hardest in NP—if any NP-complete problem admits a polynomial-time algorithm, then P = NP. Examples include the Hamiltonian cycle problem and games like Candy Crush. Most researchers believe P ≠ NP, meaning some problems are inherently hard to solve despite easy verification, though this remains unproven. The Clay Mathematics Institute guarantees the $1 million prize regardless of whether P equals NP or not.

This comprehensive exploration covers computational complexity theory. Big O notation quantifies algorithm efficiency: constant (O(1)), linear (O(n)), quadratic (O(n²)), log-linear (O(n log n)), and factorial (O(n!)) time complexities. P contains problems solvable in polynomial time; NP contains problems verifiable in polynomial time. Since solving implies verification, P ⊆ NP. NP-hard problems are at least as hard as NP-complete problems; Boolean satisfiability is NP-hard. NP-complete problems are both NP-hard and in NP (Sudoku is NP-complete). The halting problem is NP-hard but not NP-complete. The P vs NP problem asks if every efficiently verifiable problem is also efficiently solvable. Most computer scientists believe P ≠ NP, which underpins modern cryptography like RSA encryption. If P = NP, efficient factorization algorithms would break RSA. This million-dollar Millennium Prize problem remains unsolved despite decades of research.

The P versus NP problem is a fundamental unsolved question in computer science asking whether every problem whose solution can be quickly verified (NP) can also be quickly solved (P); problems in P can be solved in polynomial time, while NP problems can have their solutions verified in polynomial time, and NP-complete problems are the hardest in NP such that if any NP-complete problem can be solved efficiently, all NP problems can be, making the resolution of this problem one of the seven Millennium Prize Problems worth $1 million.

The P vs NP problem asks whether every problem whose solution can be verified quickly (NP) can also be solved quickly (P); problems in P have polynomial-time algorithms, while NP-complete problems like the Traveling Salesman Problem have no known efficient solutions but can be verified efficiently, and this remains one of the most important unsolved questions in computer science.

The P versus NP problem explores whether problems that are easy to verify (NP) are also easy to solve (P). Problems in P, like addition or prime checking, have efficient solutions and verifications. Problems in NP, such as jigsaw puzzles, are easy to verify but potentially hard to solve. The core question asks if verifying a solution and producing one describe the same set of problems. This distinction forms the foundation of computational complexity theory, categorizing problems based on their inherent difficulty and the efficiency of algorithms that solve them.

Noam Chomsky defined formal languages using terminal and non-terminal symbols with production rules. The Chomsky hierarchy classifies languages by complexity: regular grammars, context-sensitive grammars, and recursively enumerable grammars. Human languages occupy the recursively enumerable area, requiring complete Turing machines for parsing. Classical symbolic AI failed because it couldn't handle the messiness of reality with its rule-based approaches. Deep learning solved this by approximating arbitrary functions and discovering patterns without explicit rules, finding solutions superior to Chomsky grammars including handling exceptions.

The Chomsky hierarchy (finite state, context-free, context-sensitive, unrestricted) is based on subset relations between grammars. However, when applied to natural language, this hierarchy has limitations. Different levels of the hierarchy appear across the linguistic landscape, and no single level adequately describes the whole system. Chomsky argued that English is not a finite-state language due to discontinuous dependencies like subject-verb agreement. Yosef argued for mild context sensitivity to handle crossing dependencies. Shearer proposed flat phonology with only linear relations. These examples show that local computational solutions may not apply to the whole system.

Natural language is classified as mildly context-sensitive (Type-1) in the Chomsky hierarchy, requiring grammars with context-sensitive rules that can handle crossing dependencies found in certain languages like Swiss German, which cannot be captured by simpler Type-2 (context-free) grammars; this classification demonstrates that human language requires sophisticated syntactic rules involving memory and contextual awareness, placing it above basic finite-state grammars but below the overly powerful unrestricted Type-0 grammars.

The Chomsky hierarchy classifies grammars by complexity: finite state (simplest, strings together elements), context-free (hierarchical structures), and context-sensitive. Humans can learn all levels easily. Monkeys can learn finite state grammars (a*b*) but cannot learn context-free grammars (a^n b^n). This requires going beyond the innermost level of the hierarchy. The speaker uses artificial grammar learning experiments to test what animals can and cannot do.

Human language exhibits infinite recursion—the capacity to generate infinitely long grammatical strings—revealing underlying computational processes beyond mere string manipulation. The Chomsky hierarchy classifies formal languages into four types based on computational power: Type 3 (regular grammars) with finite state automata, Type 2 (context-free grammars) with pushdown automata, Type 1 (context-sensitive grammars) with linear bounded automata, and Type 0 (unrestricted grammars) with Turing machines. A formal grammar consists of terminal symbols (actual words), non-terminal symbols (derivation intermediates), a start symbol, and production rules. Context-free grammars allow rule application regardless of context, while context-sensitive grammars impose contextual conditions. The conjecture that context-free grammars suffice for human language syntax was proven false by Dutch and Swiss German examples requiring context-sensitive grammars, demonstrating that natural languages exhibit computational properties beyond context-free systems.