Basics of Probabilistic Context-Free Grammars (PCFGs) | NLP

Probabilistic Context-Free Grammars (PCFG) extend standard grammars by associating each rule with a probability, estimated from treebanks like the Penn Treebank containing approximately one million annotated words. In probabilistic CKY, each chart cell stores both non-terminals and their probabilities, computed as products of constituent probabilities and rule probabilities. The Viterbi algorithm modification discards lower-probability alternatives at each step since higher-level probabilities depend only on constituent parts, efficiently finding the single most probable parse tree. Standard PCFGs suffer from the independence assumption—rule probabilities depend only on non-terminal labels, ignoring specific words. Lexicalization creates separate rules for different head words but creates sparsity problems for rare words. The inside-outside algorithm provides iterative parameter estimation without treebanks by generating pseudo-treebanks from random initializations and improving estimates until convergence.

The inside algorithm calculates sentence probability by summing probabilities of all possible parse trees. When labeled data is available, rule probabilities are computed by counting rule occurrences. When labeled data is unavailable, the task becomes learning PCFG parameters from unlabelled sentences. The EM algorithm iterates between E-step (computing expected rule counts using inside and outside probabilities) and M-step (updating rule probabilities by dividing expected counts by total expected counts for each non-terminal). Inside probability α(A, W) is the probability that non-terminal A derives word sequence W. Outside probability β(A, W) is the probability that starting from the start symbol S, one can derive the sequence W before and after A.

Probabilistic Context-Free Grammars (PCFGs) can be learned from tree banks by extracting rules directly from annotated parse trees and estimating rule probabilities using maximum likelihood estimation, where the probability of a rule α→β equals the count of that rule divided by the count of all rules with α as the left-hand side; however, for a PCFG to define a proper probability distribution over all possible parse trees, the rule probabilities for each non-terminal must sum to exactly one, though additional conditions are required to ensure the grammar doesn't assign zero probability to all finite trees or fail to terminate with probability one.

Treebanks are annotated corpora where linguists have manually drawn syntactic trees for thousands of sentences, marking their grammatical categories. These resources enable estimation of production rule probabilities in PCFGs. By counting how often each rule appears in the treebank relative to its left-hand side occurrences, one can estimate rule probabilities using maximum likelihood estimation. For example, if rule NP → Det N appears twice out of five occurrences of Det N on the left, its estimated probability would be 0.4.

Grammar induction requires balancing unsupervised and supervised approaches. Fully unsupervised learning produces poor results; fully supervised demands extensive human annotation. Expectation Maximization (EM) provides a middle ground through iterative refinement. Starting with overly general models, EM alternates between computing expected tree usage (inside-outside algorithm) and reassessing parameters with smoothing to prevent overfitting. Comparative experiments against trigrams and PCFGs on Wall Street Journal corpus showed PLTIG performs well for both language modeling and parsing, achieving intermediate performance between trigrams (better for language modeling) and PCFGs (better for parsing). Running time advantages favor PLTIG due to lexical representation efficiency. Evaluation uses crossing brackets metric since induced trees differ from bracketed annotations.

Dependency parsing is the opposite of constituency parsing—it establishes direct relationships between individual words rather than dividing sentences into phrases. A head word is the most important word whose removal destroys the phrase's meaning, while dependents provide additional information. The root word (typically the main verb) connects to all other words through directed arcs. Each arc carries a tag indicating the grammatical relationship, such as adjective modifier (amod) or determiner (det). A key principle: heads can exist without dependents, but dependents lose meaning without their heads.

Dependency parsing is a formalism for syntactic analysis that represents sentence structure as directed binary grammatical relations between words, where each word has exactly one head (central organizing word) and can have multiple dependents (modifiers, arguments). Unlike constituency parsing which uses phrasal constituents, dependency parsing focuses on word-to-word connections labeled with grammatical relations like subject, object, and modifier. The Universal Dependencies project provides standardized dependency relations across 130+ languages. Two main parsing approaches exist: transition-based parsing (linear time, greedy, produces only projective trees) and graph-based parsing (quadratic time, finds optimal spanning tree, can handle non-projective structures). Evaluation uses Unlabeled Attachment Score (correct heads) and Labeled Attachment Score (correct heads and labels).

Dependency parsing is a syntactic formalism that defines sentence structure through direct relationships between words rather than hierarchical constituency. Each constituent has a head word—the most important element—which other words modify or depend on. Unlike constituency parsing, which uses non-terminal symbols like NP and VP, dependency parsing creates a directed acyclic graph where each word (except the root) has exactly one parent. This approach maintains equivalent structural information through subtrees while offering more direct insight into predicate-argument relationships and grammatical roles.

Dependency parsing analyzes sentence structure by identifying directed binary relationships between words, where each word (except the root) has exactly one head and may have multiple dependents, unlike constituency parsing which builds hierarchical phrase trees; the transition-based algorithm uses a stack, buffer, and set of actions (shift, left arc, right arc) to iteratively construct the dependency tree, with evaluation measured through unlabeled attachment score (correct head-child relationships) and labeled attachment score (including correct grammatical labels).

Dependency parsing represents sentence syntax through minimal trees connecting words via head-dependent relationships, where heads determine syntactic category, select dependents, and specify their form and position; unlike constituency grammars, it uses pairwise connections with an artificial root node, and parsing algorithms must produce connected, acyclic trees that are often projective (no crossing edges), though this varies across languages like Czech which frequently exhibits non-projective structures.

This section introduces constituency parsing as a method for extracting syntactic structure through hierarchical phrase trees. The CKY algorithm exemplifies this approach, using bottom-up dynamic programming to combine words into meaningful phrases like noun phrases and verb phrases. The parser reuses computational results efficiently once constituents are identified. Evaluation relies on treebanks—manually annotated datasets—with recall measuring correctly identified constituents and precision measuring correctly labeled ones. F1 score combines both metrics. A constituent is correct if it matches the reference in start position, end position, and non-terminal symbol.

Parser performance is evaluated using ParsEval metrics: precision measures the proportion of predicted constituents matching the gold standard, while recall measures the proportion of gold standard constituents correctly identified. The F-score combines both metrics. Treebanks enable cross-linguistic research as similar constituent structures (NPs, VPs, prepositional phrases) appear across languages despite variations in specific node labels. Examples include Italian, Turkish, Japanese, and Arabic treebanks. Computational linguistics uses n-ary grammar rules rather than strictly binary rules, allowing more natural representation of language structures while remaining convertible to binary formats for parsing efficiency. This theoretical agnosticism accommodates multiple linguistic frameworks without committing to any single theory.

Parsing evaluation requires supervised data with human-annotated reference parse trees (treebanks). Evaluation measures how closely the automatically generated parse matches the reference. A constituent is considered correct if it has the same start and end positions and the same non-terminal label as a constituent in the reference tree. Precision measures the proportion of constituents in the hypothesis that are correct, while recall measures the proportion of constituents in the reference that are correctly identified in the hypothesis. The F1 score combines precision and recall using the harmonic mean. These metrics were historically challenging (not reaching high 90s), but neural approaches have significantly improved performance on parsing tasks.

Parser evaluation uses two main metrics: unlabeled accuracy measures whether the parser correctly identifies which word depends on which other word, regardless of grammatical relation labels. Labeled accuracy requires both correct dependency pairs and correct grammatical relation labels (such as subject, object, noun modifier). These metrics compare parser output against gold-standard treebank annotations to assess parsing quality.

This comprehensive section presents neural methods for both constituency and dependency parsing. For constituency parsing: Neural CRF parsing uses feed-forward networks to score spans then applies CKY and inside-outside algorithms for structured inference; simpler span classification distinguishes constituents from non-constituents then reconstructs trees via CKY or top-down algorithms; modern approaches use self-attention models like the Berkeley parser. For dependency parsing: Graph-based approaches represent sentences as fully connected directed graphs where edges represent potential dependencies; the Chu-Liu-Edmonds algorithm finds maximum spanning trees; biaffine scoring calculates arc scores using bilinear correspondence between head and dependent vectors with separate representations for heads and dependents.

Neural constituency parsing assigns scores to all possible parse trees using multi-layer perceptron span classifiers with bidirectional LSTM features, then employs the CKY algorithm with empty labels for efficient global optimum inference, where the recursive span law enables bottom-up construction of higher-level spans from lower-level ones, achieving O(n²L²) complexity without explicit grammar rules.

Dependency parsing analyzes sentence structure by identifying head words and their dependent modifiers, forming a tree-like structure where each word (except the root) has exactly one parent. Unlike constituency grammars that group words into hierarchical constituents, dependency grammars focus on asymmetric relationships between heads and modifiers. Modern neural dependency parsers use transition-based or graph-based approaches, where neural networks predict parsing decisions based on distributed representations of words, parts of speech, and contextual features. Transition-based parsing uses a buffer and stack with shift/reduce operations, while graph-based parsing scores all possible dependencies and selects the minimum spanning tree. Neural approaches with word embeddings and deep learning have achieved state-of-the-art performance, achieving over 94% unlabeled attachment score on benchmark datasets.

Deep learning approaches have been successfully applied to syntactic parsing tasks. Neural network methods for dependency parsing have proven highly effective, with techniques developed at Stanford being used in Google's Parsey McParseface and SyntaxNet systems. These neural parsers automatically learn to identify the grammatical structure of sentences without relying on manually crafted grammatical rules.

Dependency parsing analyzes sentence structure by identifying modification relationships between words, where each word (except the root) has exactly one head word it depends on; this approach, which emphasizes direct modification links over nested constituency structures, has become dominant in modern NLP since around 2000 due to its effectiveness with machine learning methods and cross-linguistic applicability through frameworks like Universal Dependencies.