How to Create Your Own Academic Corpus with AntConc

Parts of speech tagging identifies grammatical roles of words using standardized tags: NN (noun), VBD (past tense verb), JJ (adjective), RB (adverb), DT (determiner), IN (preposition). This enables statistical analysis, category-based evaluation, and semantic inference. Common tags include NNP (proper noun), VBG (gerund), JJR (comparative adjective), RBR (comparative adverb). POS tagging helps distinguish between homonyms and understand contextual usage of words.

POS tag annotation is the process of attaching grammatical labels (tags) to tokens in a corpus, enabling efficient search and analysis of linguistic patterns. Unlike manual keyword searching, annotated corpora allow researchers to query entire grammatical categories (e.g., all past tense verbs) using concise tag codes, significantly simplifying corpus analysis. Different annotation schemes (such as CLAWS and Stanford POS) use varying formats and tag sets, but all serve the same purpose of enabling systematic linguistic research through automated grammatical classification.

Part-of-speech tagging assigns grammatical categories to words, distinguishing their roles in sentences (e.g., 'run' as base verb vs. infinitive). This essential annotation supports lexicography, parsing, and many downstream applications. Tagging precision varies by language morphology and depends on training data quality, achieving ~90-95% accuracy for English through statistical methods. Syntactic parsing analyzes sentence structure, evolving from manual annotation to statistical approaches that assign probable analyses based on training corpora. Results appear as phrase structures, dependency trees, or constituency trees. Manual post-processing corrects automatic errors. Major annotated corpora like the British National Corpus (1M words, 60% spoken) demonstrate these techniques, with syntactic annotations typically following part-of-speech tagging.

Part-of-speech tagging assigns grammatical labels (such as NN for nouns, VV for verbs, JJ for adjectives) to each word in a corpus, enabling researchers to search for specific word classes and their inflected forms using tools like LancsBox and TagAnt/AntConc; this involves using regular expressions (asterisks for zero or more characters, vertical bars for alternatives) to efficiently locate linguistic patterns across large text collections.

Syntactic annotation identifies grammatical structure (subjects, objects, functions) enabling searches by grammatical relationships. Semantic role annotation identifies participant roles (agent, patient, instrument) in events described by sentences. This enables understanding of meaning and development of AI systems for question-answering. For example, in 'I eat chocolate,' 'I' is the agent and 'chocolate' is the patient. These annotations support sophisticated linguistic analysis and AI applications requiring deep semantic understanding.

The keyword function in the British National Corpus (BNC) compares the frequency distribution of a source corpus against a reference corpus to identify statistically significant differences, allowing researchers to discover distinctive language features such as pronouns and conversation fillers that characterize spoken versus written language.

This section examines practical research implications of corpus design. COCA contains approximately 485,517 texts while BNC contains only 4,540 texts (less than one-hundredth as many), dramatically impacting virtual corpus creation—the ability to create focused subsets based on words, phrases, or metadata. With only 4,000 texts in BNC, many words or phrases lack sufficient related texts for meaningful analysis. Virtual corpus examples demonstrate this: searching for 'investment' yields much better quality keywords in COCA. The section then addresses fundamental limitations of the BNC as a historical corpus: it contains no texts from the 20 years between its two snapshots (1994 and 2014), making it impossible to map linguistic, cultural, or societal changes. COCA provides continuous data from 1990 to 2019 with at least 20 million words each year and consistent genre balance, enabling precise tracking of changes. Examples include 'Afghanistan' increasing 20-fold after the 2001 US invasion, and 'Y2K' appearing only in the late 1990s. COCA also enables collocation analysis over time, showing how the collocations of 'crisis' shifted from the early 1990s to the 2010s.

Corpus reference represents one extreme of corpus construction, while specialized small corpora represent another. Corpus reference serves as a foundation for general linguistic studies across language history, requiring careful balance between size, cost, coding depth, and design. At the opposite extreme are highly specialized corpora with very specific objectives and extremely detailed coding, such as the medieval Bible project which aligns Hebrew, Latin, and Spanish translations with manuscript images and critical editions. Each type serves different research needs effectively.

The Keywords tab identifies words significantly more frequent in one corpus compared to another using the log likelihood test. Users compare target corpora (e.g., quotes) against reference corpora (e.g., non-quotes). Results reveal dialogue markers (first-person pronouns, contractions) in speech and narration markers (reporting verbs) in non-quotes. The n-gram option creates key clusters showing significant phrase differences. Results can be swapped to view the opposite perspective.

Keyword analysis is a corpus linguistics technique that identifies words appearing unusually frequently or infrequently in one corpus compared to another, using statistical measures like log likelihood to reveal distinctive vocabulary patterns that indicate cultural, ideological, or stylistic characteristics of specialized discourse; this method requires comparing a specialized corpus with a reference corpus and can be enhanced by combining it with concordance and dispersion plot analyses for deeper textual understanding.

Statistical association measures identify meaningful collocations by considering frequency and other criteria. Expected frequency establishes a random co-occurrence baseline by multiplying node frequency by collocate frequency and dividing by corpus size. The M-score compares observed to expected frequency using a logarithmic scale, indicating how many times more frequently a collocation appears than random chance would predict. Different association measures highlight different aspects: M-score emphasizes rare exclusivity; M3 compensates for frequency bias; LogDice and M2 focus on exclusive term extraction; T-score and Z-score relate to frequency; Delta P identifies directionality (asymmetry in collocation relationships). Collocation networks extend analysis beyond first-order collocations by treating collocates as new nodes for further analysis, creating associative chains where meaning 'cross-bleeds' through the network.

Three statistical measures rank collocations by strength: (1) Mutual Information (MI) - prioritizes rare but strongly associated pairs; (2) Log-Likelihood (LL) - favors frequent associations; (3) Chi-Square (χ²) - balances frequency and association strength. These measures appear in corpus interfaces when searching for collocations. Different measures yield different rankings - MI might prioritize 'cuchara de madera' (spoon of wood) while LL might favor more common combinations. Researchers should understand that no single measure is universally best; choice depends on research goals. Manual verification of top-ranked candidates is recommended.

This section covers collocation analysis for identifying frequently co-occurring words within specified context windows. Results are visualized as networks where nodes represent words and edges represent collocation relationships. Collocation strength is measured using Log-Likelihood, Mutual Information, and Dice coefficient. Higher values indicate stronger associations. Visualization plots collocations on axes showing frequency and association strength. Node size represents frequency, edge length represents association strength, and node position indicates distribution patterns. Users can add multiple target words to compare collocation patterns and identify shared collocations across related terms.

Colocation refers to word combinations occurring more frequently than by chance. Association measures quantify syntactic connection strength between components. The Louch Dice measure, independent of corpus size, allows cross-corpus comparison. A colocation is statistically significant when Louch Dice > 0, with maximum value 1.414. This provides objective information about which words form stable combinations and their association strength.

This section presents the coefficient of colligation as a measure of association strength between two attributes. Students learn the complete formula: Q = [(AB × αβ) - (Aβ × αB)] / [(AB + Aβ)(Aβ + αβ)(AB + αB)(Aβ + αB)]. The section emphasizes systematic application: first complete the contingency table, then substitute values into the formula. The numerator captures the difference between positive and negative associations, while the denominator normalizes the result for interpretation.

This section provides practical NLP implementation using Python libraries. With NLTK, tokenization splits text into meaningful units, POS tagging identifies grammatical categories, stemming reduces words to root forms, lemmatization returns valid dictionary words, and stop word removal filters common words. Vectorization techniques like Bag of Words and TF-IDF convert text to numerical representations. SpaCy provides advanced tools: Named Entity Recognition identifies entities like organizations and dates, POS tagging identifies grammatical categories, and dependency parsing analyzes grammatical relationships between words. Text classification assigns categories to text using pre-trained models or custom machine learning approaches. These hands-on techniques demonstrate how NLP transforms unstructured text into machine-readable formats for analysis.

Natural Language Processing (NLP) is the process of enabling computers to understand natural language by converting text into numerical data that machines can process; the Natural Language Toolkit (NLTK) for Python provides essential tools for this task, including tokenizers that split text into words and sentences, corpora which are large bodies of text for analysis, and lexicons which are dictionaries of words and their meanings, with NLTK offering both basic and advanced tokenization methods that handle complex cases like abbreviations and punctuation that simple string operations cannot manage.

spaCy is a powerful NLP library that processes text through a pipeline to perform common text analysis tasks including tokenization (breaking text into individual words/punctuation), lemmatization (reducing words to their root/base form), part-of-speech tagging (identifying grammatical roles like nouns and verbs), named entity recognition (identifying people, organizations, locations, dates), and dependency parsing (analyzing grammatical relationships between words), all returning a document object where each token contains rich linguistic information.

This section demonstrates spaCy's core text processing capabilities: sentence segmentation breaks text into meaningful units; named entity recognition identifies persons, organizations, locations, and dates; part-of-speech tagging assigns grammatical categories; lemmatization reduces words to base forms; token shape analysis captures word characteristics; and token probability measures word frequency. spaCy's multi-threading capability (releasing Python's GIL) enables efficient parallel processing of large text corpora.

This lecture introduces two essential NLP libraries—NLTK (built for research) and SpaCy (built for industrial applications)—and demonstrates core text processing techniques including tokenization (breaking text into words/sentences), stop word removal, part-of-speech tagging, chunking, lemmatization/stemming, and frequency distribution analysis. The video emphasizes that tokenization is application-dependent, requiring different approaches for financial data versus social media text, and explains how libraries like spaCy enable dependency parsing and named entity recognition for information extraction.