Lexer Explained: From Characters to Tokens

Regular expressions (regex) are powerful text pattern matching tools that allow users to search, filter, and manipulate text efficiently using special symbols like carrots (^) for line beginnings, periods (.) as wildcards, dollar signs ($) for line endings, pipes (|) for command chaining, and curly braces {} for repetition control; these patterns can be applied across various Linux utilities like grep and find to extract specific information from large text datasets.

Regular expressions are patterns used to define how tokens should appear in text, enabling string matching, searching, text manipulation, data validation, and text processing. Pattern matching identifies whether specific words, numbers, or patterns exist within text. Examples include digit matching (0-9), date format validation (DD/MM/YYYY), email address matching, and Indian phone number validation (+91 6-9 followed by 9 digits). These patterns ensure text conforms to expected formats by defining acceptable character ranges and structures.

Regular expressions are powerful tools for extracting information from text sources like code, log files, spreadsheets, and documents. Everything in regex is essentially a character, and users write patterns to match specific sequences of characters (strings). Most patterns use ASCII characters (letters, digits, punctuation), while Unicode supports international text. Visual tools highlight matching characters as patterns are typed, making learning intuitive. Patterns must match exact sequences, not just contain characters. The goal is to write patterns that match across multiple lines by identifying common characters. For example, 'ABC' matches all three rows because it contains characters common to each line, while 'D' or 'F' match fewer rows based on character frequency.

Regular expressions (regex) are powerful text pattern-matching tools used across programming languages, SQL, KQL, and command-line utilities. They enable three core operations: checking if a pattern exists in a string, extracting specific elements from text, and replacing matched text with new content. Regex is essential for data validation, log file analysis, website scraping, and text transformation tasks. Understanding regex syntax enables efficient text processing in nearly any development or data analysis environment.

Regular expressions (regex) are patterns used for text matching that are useful across many contexts including Unix shells, programming languages, and data analysis tools. They allow you to search for and match specific patterns in text. The regex101.com website provides an interactive tool for testing regular expressions, allowing you to see how patterns match against test strings before using them in actual commands.

Compilers transform human-readable source code into machine-executable binaries through a systematic pipeline. Hardware understands only binary instructions, so compilers translate high-level code into this format. The complete tool chain consists of four stages: Preprocessor converts source code to .i files, Compiler generates assembly language, Assembler creates object files (.o), and Linker combines them into the final executable. Modern compilers are organized into three distinct stages: Front End analyzes source code through syntax parsing, semantic analysis, type checking, and scope resolution to generate Intermediate Representation (IR). The Middle End performs optimization passes on IR to improve performance while preserving correctness. The Back End translates optimized IR into target-specific machine code through lowering, instruction selection, and code generation. This modular architecture enables independent development of each component, allows reuse across different source languages, and separates target-independent optimizations from architecture-specific code generation.

High-level programming languages like C are translated through a compilation pipeline into executable machine code. The process begins with source code containing expressions like A + B + C + D × E, which the compiler converts into assembly instructions. These assembly instructions represent low-level operations the CPU can execute directly. The resulting machine code consists of multiple instructions: loading values from memory into CPU registers, performing arithmetic operations using the ALU, and storing results back to memory. This translation enables human-readable code to interface with the binary instructions that CPUs understand.

The compilation pipeline consists of four phases that transform high-level code into executable programs: (1) Preprocessing - modifies code with directives starting with # (like #include), copies header files, and deletes comments; (2) Compilation - translates preprocessed code into assembly language (.s file); (3) Assembly - converts assembly into relocatable object code (.o file) containing machine instructions; (4) Linking - combines object files and resolves external symbol references to produce a final executable. Static linking copies library functions directly into the executable, while dynamic linking uses pointers to load libraries at runtime.

This section explains the complete compilation pipeline used to transform high-level source code into executable programs. The process involves multiple stages: source code is first compiled into assembly language, a readable textual format describing machine instructions. An assembler then converts this assembly code into binary machine language. When programs use external libraries, a linker combines the compiled program code with previously compiled library routines to produce a complete executable. Kathleen Booth invented the first assembly language for the ARC computing system, enabling low-level programming. Additionally, preprocessors like those in C allow conditional inclusion/exclusion of code sections based on configuration parameters, enabling single-source compilation across different platforms.

ASCII is a 7-bit character encoding system that represents 128 characters using binary values, originally designed for English text but limited in scope; Unicode is a universal character encoding standard that supports virtually all written languages and symbols worldwide by assigning each character a unique hexadecimal code point; UTF-8 is a variable-length encoding algorithm that implements Unicode by representing common ASCII characters with just one byte while using up to four bytes for less common characters, making it backward compatible with ASCII and now accounting for approximately 98% of web pages.

Strings in programming are sequences of bytes that require three components: size (memory length), charset (character mapping), and encoding (byte interpretation). ASCII uses 7 bits per character (128 total), while Unicode provides a universal character set. UTF-8 is an encoding algorithm for Unicode that uses variable-length bytes (1-4 bytes per character), making it backward compatible with ASCII. Characters like Chinese characters require multiple bytes (32 bits/4 bytes per character), which is why string indexing can corrupt encoding if not handled properly. This is why most programming languages make strings immutable to prevent encoding corruption.

Unicode is a comprehensive character encoding standard that maps human-readable graphemes (writing system units) to one or more code points (numeric values), which are then encoded into binary using schemes like UTF-8 (variable-length, 1-4 bytes, backward-compatible with ASCII), UTF-32 (fixed 4-byte encoding), or others; unlike ASCII where one character equals one byte, Unicode requires understanding that graphemes, code points, and bytes are distinct concepts, and using byte-level string manipulation on Unicode data can corrupt text, necessitating proper encoding awareness for correct string handling.

Character encoding determines how characters are represented in computers. ASCII (1963) used 7 bits for 128 characters, sufficient for English but lacking accents. Regional code pages emerged by using the 8th bit, doubling capacity to 256 characters, but failed to accommodate all human languages. Mojibake (character corruption) occurred when text encoded in one encoding was read with another. Unicode was created in 1988 to solve this by providing a single universal standard supporting over 170 writing systems and 150,000+ characters, assigning each character a unique code point (e.g., U+0041 for 'A'). UTF-8 (Unicode Transformation Format - 8) converts code points to binary using variable width: 1 byte for ASCII, 2 bytes for Latin Extended/Greek/Cyrillic, 3 bytes for East Asian characters, and 4 bytes for emojis. It uses bit prefixes to indicate character size and maintains ASCII backward compatibility. Endianness determines byte order for multi-byte encodings, while BOM indicates byte order for fixed-width encodings but should never be used with UTF-8.

Strings are sequences of characters represented by integer codes through encoding systems. ASCII assigns: uppercase A-Z (65-90), lowercase a-z (97-122), digits 0-9 (48-57), space is 32. Unicode is a superset supporting global characters including emojis. Python provides ord() to convert characters to integers and chr() to convert integers to characters. Understanding these encoding principles enables low-level string manipulation and character-based algorithms.