AntConc 4.2 Tutorial: Getting Started with Corpus Analysis

Text annotation encodes linguistic analysis into corpus data. Part-of-speech tagging assigns grammatical categories using standardized tags. Parsing analyzes sentence structure by identifying grammatical relationships. Semantico performs semantic annotation by categorizing words into semantic fields. Annotated corpora enable searches by linguistic tags and precise feature analysis. The British National Corpus (BNC) is a modern corpus of British English with balanced genre representation, though not continuously updated. Annotated corpora allow researchers to study specific grammatical features like passive voice by identifying past participles, which would be impossible with unannotated data.

Corpus annotation transforms raw text into powerful analytical tools by adding interpretive linguistic information. This process includes phonological annotations for pronunciation, syntactic annotations for grammatical structure, semantic annotations for meaning, and discourse annotations for conversation flow. Part-of-speech tagging assigns standardized codes like NN1 (singular common noun), VV (base form verb), and JJ (adjective) to each word, enabling computational analysis. Researchers can query all nouns by searching 'NN1' or examine verb behavior by querying verb tags. While software automates much of this tagging, manual verification remains necessary for accuracy. Annotated corpora differ fundamentally from pure corpora, with linguists holding varying preferences between these approaches based on research objectives and analytical needs.

Part-of-speech tagging assigns grammatical labels (such as NN for nouns, VV for verbs, JJ for adjectives) to each word in a corpus, enabling researchers to search for specific word classes and their inflected forms using tools like LancsBox and TagAnt/AntConc; this involves using regular expressions (asterisks for zero or more characters, vertical bars for alternatives) to efficiently locate linguistic patterns across large text collections.

Semantic tag annotation is a corpus linguistics method that categorizes words by their meaning rather than grammatical function, enabling automated content analysis, semantic extraction, and comparative studies across sub-corpora; in CQPweb, semantic queries use square brackets with semantic codes (e.g., [semantik kg='A1.1.12'] for time-related words) to retrieve words within specific semantic categories.

Annotasi is the process of attaching analysis labels to tokens, analogous to adding toppings to bread. It involves creating tag sets with human-designed codes (e.g., D for determiners, N for nouns, V for verbs). Types include morphosyntactic (POS tagging), lemmatization, syntactic, semantic, morphological, pragmatic, and discourse annotasi. Different schemes use varying formats and labels: Claus uses 'NN1' for nouns with identical singular/plural forms and 'NN2' for different forms, while Stanford POS uses 'NN' for both. Understanding these variations is essential for selecting appropriate corpora and interpreting search results correctly.

A fundamental problem in corpus linguistics is that traditional association measures confound multiple dimensions that should be measured independently. Log likelihood values, the most frequently used association measure, are extremely strongly correlated with frequency (explaining approximately 95% of variance), meaning they essentially just transform frequency rather than measuring genuine association. Similarly, most dispersion measures are highly correlated with frequency (r > 0.8). This creates a situation where researchers think they are measuring association or dispersion, but are actually just measuring frequency in a slightly different form. This confounding prevents proper cognitive and psycholinguistic analysis because these three dimensions—frequency, association strength, and dispersion—actually matter separately for understanding language acquisition, processing, and change.

Statistical association measures identify meaningful collocations by considering frequency and other criteria. Expected frequency establishes a random co-occurrence baseline by multiplying node frequency by collocate frequency and dividing by corpus size. The M-score compares observed to expected frequency using a logarithmic scale, indicating how many times more frequently a collocation appears than random chance would predict. Different association measures highlight different aspects: M-score emphasizes rare exclusivity; M3 compensates for frequency bias; LogDice and M2 focus on exclusive term extraction; T-score and Z-score relate to frequency; Delta P identifies directionality (asymmetry in collocation relationships). Collocation networks extend analysis beyond first-order collocations by treating collocates as new nodes for further analysis, creating associative chains where meaning 'cross-bleeds' through the network.

Collocates analysis identifies words that frequently appear near target words, revealing linguistic associations. AntConc's Collocates tool uses adjustable parameters including window span (words to search left/right) and minimum frequency thresholds. The default ranking uses Mutual Information (MI) score, which measures exclusivity - words strongly associated with the target despite low absolute frequency. MI prioritizes statistical significance over mere frequency, meaning a word appearing only 3 times with strong exclusivity may rank higher than one appearing 22 times with weak exclusivity. Researchers can alternatively sort by raw frequency for different analytical perspectives.

Collocation analysis uses statistical measures to identify strong associations between words. Mutual Information (MI) is commonly used to identify collocations with strong lexical associations. Different statistical measures (like chi-square) will produce different lists of collocations. Researchers should choose measures appropriate for their analysis goals, as some measures favor content words (nouns, verbs, adjectives) while others may highlight function words.

Log-likelihood is a statistical measure for collocation strength that compares observed word combinations against chance expectations. Higher scores indicate stronger collocations. In the example, 'chips' scores 2378 with 'fish' while 'meet' scores 941, confirming native speaker intuition. Lemmatization groups different word forms (singular, plural, tenses) under single lemmas, combining their frequencies and scores. When enabled, 'chips' and 'chip' merge, producing a combined score of 3055 (higher than 2000 for 'chips' alone). This provides more comprehensive collocation analysis across grammatical forms.

Reference corpora require prior design involving decisions about objectives, target audience, and distribution parameters. This includes determining what percentage of texts should correspond to each year, country, and text type (press, narrative, oral, etc.). The Corpus del Español del Siglo XXI (Corpes) exemplifies this approach, with 25 million word forms per year, 30% from Spain and 70% from America, plus Guinea Ecuatorial and Philippines. The goal is representativeness at the level of linguistic zones (like Rioplatense) and quinquennial periods, rather than year-by-year representation. Representativeness in corpus linguistics differs fundamentally from survey sampling. In surveys, researchers aim for samples that mirror the population distribution. However, in linguistics, the 'universe' of language use is unknown, making true representativeness impossible. Instead, corpus linguists use 'balance' or 'equilibrium' - ensuring that the percentage of texts from each group and subgroup is considered appropriate for the research goals.

In corpus linguistics, effective corpus design begins with clearly defining the research purpose, followed by an iterative process involving pilot investigation, corpus design using a sampling frame, compilation, investigation, and refinement; representativeness can be approached in two ways—genre-based representativeness (balancing different text types within a domain) and linguistic representativeness (matching overall language patterns through statistical analysis of linguistic features like noun frequency per thousand words with standard deviation calculations)—with different corpus sizes required for different features (smaller corpora suffice for common grammatical features, larger ones for common lexical items, and very large corpora for rare lexical items).

A common mistake in corpus design is assuming that simply collecting more data will eliminate bias. However, if the sampling method itself is biased (such as collecting only from Twitter), the resulting corpus will reflect that bias regardless of its size. The BCCWJ project demonstrates that careful, systematic sampling based on actual distribution data is essential for creating a truly balanced corpus.

When compiling corpora, two fundamental principles must be followed: (1) Representativeness - the collected texts should represent a particular language, language variety, register, or genre accurately; and (2) Balance - the corpus should provide proportional representation across different aspects of the target language or variety. These principles ensure that the corpus serves as a reliable source for linguistic analysis and research.

Creating a valid corpus requires following established principles: (1) Representativeness - the corpus must include diverse text types, genres, and contexts that adequately represent the language; (2) Balance - text types should be proportionally distributed to avoid skewing results; (3) Temporal distribution - texts should be evenly distributed across time periods; (4) Text type diversity - including both written and spoken language, different genres, and various communication contexts. The speaker emphasizes that a corpus with only newspaper articles cannot represent literary language, and a corpus with only novels cannot represent everyday communication. The goal is to create a corpus that can yield generalizable linguistic findings.