U-Net Architecture Explained: Encoder-Decoder for Image Segmentation

Denoising Diffusion Probabilistic Models (DDPM) represent a powerful class of latent variable models for generating high-quality synthetic images, underpinning advanced systems like DALL·E 2 and Imagen. The framework employs two interconnected Markov chains: the forward diffusion process gradually adds Gaussian noise according to a variance schedule β₁ to βₜ, progressively destroying image signals until only pure noise remains; the reverse process reconstructs realistic images by iteratively removing noise from diagonal Gaussians. Mathematically, alpha parameters (α = 1-β, ᾱₜ as cumulative products) enable closed-form sampling at any time point, facilitating efficient training through stochastic gradient descent. The reverse process uses a UNet backbone to predict and remove noise, with training involving uniform time-point sampling, noise injection, and parameter optimization to minimize negative log likelihood. Evaluation relies on Inception Score (measuring sharpness and diversity) and FID (comparing feature distributions via Inception V3), achieving state-of-the-art results on CIFAR-10 with FID 3.17 and Inception Score 9.46.

The UNet architecture serves as the backbone for diffusion models, featuring an encoder-decoder structure with hierarchical processing. Down blocks progressively reduce spatial dimensions while increasing channel depth, incorporating ResNet blocks with residual connections and self-attention mechanisms. Each ResNet block contains two convolutional layers with group normalization and activation functions, connected by residual connections. Following this, self-attention layers process spatial relationships across all pixels. Mid blocks maintain constant resolution through stacked ResNet and attention layers. Up blocks reverse the downsampling process using transposed convolutions, incorporating skip connections from corresponding down blocks. Time embeddings are fused through projection layers in each ResNet block. The complete UNet class integrates all components: initial convolutions match input dimensions, down blocks progressively refine features, mid blocks maintain resolution, and up blocks restore spatial dimensions while reducing channels. Final normalization and convolution return outputs to input dimensions, enabling the model to predict noise at each diffusion time step.

U-Net serves as the core neural network architecture for diffusion models, featuring encoder-decoder structure with skip connections that preserve spatial information. Positional encodings inject timestamp information to guide noise removal intensity. Diffusion models offer significant advantages over alternatives like GANs: they use simple mean square error objectives for stable training, employ iterative noise removal for better image quality, and avoid mode collapse common in GANs. By predicting noise incrementally rather than generating final images directly, diffusion models produce more diverse outputs while maintaining computational feasibility.

DDPM (Denoising Diffusion Probabilistic Models) consists of two fundamental processes: the forward process gradually adds noise to an image following a specific schedule until it reaches a Gaussian distribution, while the reverse process denoises from pure noise to reconstruct the image. The forward process q(xt|xt-1) is a Gaussian distribution with mean √(1-βt)×xt-1 and covariance βt×I. The reverse process is approximated by a neural network that predicts the added noise. Training minimizes the Variational Lower Bound, which simplifies to minimizing KL divergence between Gaussian distributions. Jonathan Ho empirically found that predicting the added noise works better than the original ELBO formulation. The UNet architecture consists of a downsampling path followed by an upsampling path, similar to an autoencoder. Since parameters are shared across time steps, time encoding using sinusoidal position embeddings (similar to Transformers) helps the network know at which time step it is operating. Attention modules are added between convolutional blocks, performing QKV operations like in Transformers. Group normalization is used for normalization.

Denoising Diffusion Probabilistic Models (DDPM) generate new data samples by learning to reverse a noise addition process: first, images are gradually corrupted by adding Gaussian noise until they become pure noise; then, a neural network is trained to remove this noise step-by-step, transforming random noise back into meaningful images. The training objective simplifies from maximizing the likelihood of generating real samples to minimizing the mean squared error between predicted and actual noise at each diffusion step, making the method computationally tractable while achieving state-of-the-art image generation quality.

Medical imaging AI development follows an end-to-end workflow spanning data acquisition, annotation, model training, and clinical deployment. MONAI addresses each stage through specialized components: data access via wrappers for TCIA and Medical Decathlon datasets; annotation through MONAI Label with viewer integrations; training through MONAI Core with GPU-optimized implementations; and deployment through MONAI Deploy using containerization for reproducibility. Hollow Scan extends this workflow for real-time applications requiring millisecond-level inference for procedures like endoscopy. The Model Zoo provides pre-trained models across CT, MRI, pathology, and endoscopy modalities for segmentation, detection, and reconstruction tasks. This integrated ecosystem enables researchers to progress from raw data through trained models to clinical deployment, supported by extensive tutorials and community resources.

MONAI enables rapid implementation of complete supervised training pipelines in approximately ten lines of code, demonstrating the framework's efficiency. The workflow includes importing tools, composing transformations, defining datasets with download specifications, setting up trainers with network parameters, optimizers, loss functions, and inference methods. Beyond practical implementation, MONAI serves as a platform for disseminating research through reimplementation of key publications (CoupleNet for lung segmentation, LAMP for multi-GPU model parallelism). This dual approach—providing practical tools while establishing research standards—positions MONAI to become foundational to medical imaging AI similarly to how PyTorch and TensorFlow are foundational to computer vision, enabling researchers to trust tools and focus on scientific questions.

MONAI (Medical Open Network for Artificial Intelligence) is an open-source PyTorch-based framework developed by Nvidia and King's College London that accelerates the adoption of AI in healthcare by providing a comprehensive platform covering the entire AI lifecycle through four integrated modules: Model Zoo (pre-trained models for medical imaging tasks), Label (active learning for efficient data annotation), Core (training engine with optimized medical imaging capabilities including Auto 3D segmentation and federated learning), and Deploy (application packaging for clinical deployment). The framework enables critical medical AI applications including segmentation, classification, detection, registration, and reconstruction, and has been adopted by 57 medical centers across 159 countries, with over 170 research publications.

MONAI (2019) emerged from collaboration between King's College London, NVIDIA, German Cancer Research Center, and Kitware. It covers the complete medical imaging workflow: image reconstruction, segmentation using Transformer networks, registration, longitudinal studies, and survival prediction. By September 2022, it achieved 1.1M downloads with participants from 54 countries. The AI-assisted annotation system reduces expert annotation time by ~75% through intelligent dataset selection and iterative learning. Deployed via MONAI Deploy as DICOM services or embedded systems, returning results as RTSTRUCT objects for clinical integration.

This tutorial demonstrates how to train a medical image segmentation model from scratch using MONAI Label and 3D Slicer, covering the complete workflow from downloading the COVID-19 CT Lung and Infection Segmentation Dataset to preparing the dataset, setting up the MONAI Label server with the radiology app, using the 'Grow from Seeds' functionality in 3D Slicer to annotate multiple volumes, submitting labels to the server, and training the model; the recommended approach involves annotating 10-15 volumes and training for approximately 800 epochs to achieve decent lung segmentation results.

Segment Anything (SAM) is a Transformer-based foundation model for segmentation trained on 1 billion masks from diverse datasets. The key innovation is zero-shot segmentation capability - given any image and a prompt (point, bounding box, or text), the model generates the corresponding segmentation mask. The architecture consists of an image encoder (Vision Transformer), a prompt encoder (using positional encoding for points/boxes and CLIP text encoder for text), and a mask decoder (Transformer decoder with FC layers). The model always outputs three candidate masks, improving robustness. SAM achieves zero-shot segmentation performance, outperforming previous models even without training on specific datasets. It extends to LangSAM by combining with GroundingDINO for language-based segmentation.

Image segmentation partitions digital images into regions sharing characteristics. Traditional approaches were application-specific (medical imaging, pedestrian detection, satellite analysis) and faced two major challenges: expensive manual pixel-by-pixel annotation and lack of promptability (models could only detect specific trained masks). Segment Anything introduces three innovations: (1) Promptable segmentation supporting points, boxes, and text inputs; (2) Fast encoder-decoder model generating masks in ~50ms; (3) Ambiguity awareness returning multiple masks when prompts could refer to multiple objects. The Vision Transformer (ViT) processes images by dividing them into fixed-size patches, flattening into sequences, and applying linear projections. A class token is prepended and fed through self-attention, interacting with all patches to capture global image information. The final class token output predicts the image class, enabling the model to understand overall image content.

The Segment Anything Model (SAM) is a foundational segmentation model developed by Meta that achieves zero-shot performance across multiple segmentation tasks by being prompted with diverse inputs (points, boxes, masks, or text) and trained on a massive dataset of 1.1 billion masks generated through an iterative process involving human annotators and the model itself; the architecture consists of an image encoder (ViT), a prompt encoder, and a mask decoder with a two-way attention mechanism that resolves ambiguity by outputting multiple masks and selecting the best one via IOU prediction, with subsequent variants like Fast SAM and Mobile SAM achieving significant speedups through architectural simplifications and knowledge distillation.

This section provides an in-depth exploration of SAM (Segment Anything Model), the first foundation model for image segmentation. A foundation model is trained on broad data usable for different tasks with minimal fine-tuning, similar to GPT in NLP. SAM uses a prompt-based approach supporting diverse prompt types: point prompts (single point gives mask around it), point set prompts, bounding box prompts, mask prompts, and text prompts. The architecture consists of image encoder (pre-trained ViT), prompt encoder (positional encodings for points/boxes, CLIP for text, convolutions for masks), and lightweight mask decoder with Transformer decoder performing cross-attention. SAM is trained on SA-1B dataset with 1 billion masks using model-in-the-loop annotation strategy. Evaluation results include single point valid mask evaluation on 23 datasets, zero-shot text to mask capability, and zero-shot edge detection on BSDS500. SAM computes masks in amortized real-time (~50ms per mask) and is ambiguity-aware.

FastSAM, published in June 2023, addresses SAM's computational limitations by replacing the transformer-based ViT encoder with a CNN backbone (ViT-8). This architectural change reduces parameters by 10x while achieving 50x faster inference speed. The model employs a two-stage framework: first predicting all possible masks regardless of user input, then using prompt-guided selection via CLIP similarity for text prompts or IoU calculations for bounding boxes. With only 0.04 seconds per image inference time, FastSAM enables real-time applications while maintaining performance comparable to the original SAM. This demonstrates that CNN backbones can effectively replace transformer architectures for segmentation tasks.