Academic Databases
Explore Our Semantic Databases
STEM & Tech
Math, Physics, CS & Engineering...
Economics
Macro, Micro, Econometrics...
Business
Management, Marketing, Finance...
Health & Life Sciences
Biology, Medicine, Neuroscience...
Humanities & Social Sciences
Psychology, Law, Philosophy, History...
General Studies
General & interdisciplinary studies...
Explore Topics
Features

Adversarial Examples & Training in Deep Learning | Stanford Lecture 16

Added: 2026-08-07

Deep Dive

Fundamental concepts of Deep Learning, including feedforward neural networks, Convolutional Neural Networks (CNNs), and the backpropagation algorithm.

Learn Machine Learning Like a GENIUS and Not Waste Time

Learn Machine Learning Like a GENIUS and Not Waste Time

248.4K

Deep learning is a subset of machine learning using neural networks inspired by brain structure. Key concepts include: (1) Feedforward networks - simplest type where data flows in one direction from input to output; (2) Backpropagation - the algorithm that enables neural networks to learn by making predictions, calculating error, and adjusting weights to reduce error; (3) Convolutional Neural Networks (CNNs) - specifically designed for image data, detecting edges, shapes, and complex objects; (4) Recurrent Neural Networks (RNNs) - handle sequential data like text or time series by maintaining memory of previous inputs, ideal for language translation and stock price prediction.

Click to watch →

Lecture 4 | Introduction to Neural Networks

Lecture 4 | Introduction to Neural Networks

714.3K

Backpropagation is a recursive algorithm that computes gradients for arbitrarily complex functions by representing them as computational graphs and applying the chain rule backward from the output to the inputs, enabling efficient training of neural networks through gradient descent optimization.

Click to watch →

#46 Feedforward Neural Network | Machine Learning for Engineering & Science Applications

#46 Feedforward Neural Network | Machine Learning for Engineering & Science Applications

106K

A feedforward neural network is a collection of artificial neurons organized into three types of layers: the input layer (receiving feature vectors), hidden layers (processing information), and the output layer (producing predictions). Each neuron contains a linear combination of weighted inputs plus a bias, followed by a nonlinear activation function. The feedforward process computes predictions by sequentially applying these operations across layers. In a fully connected network, every neuron connects to every neuron in adjacent layers, requiring N*M + M weights for layers with N and M neurons respectively. Networks with multiple hidden layers are termed 'deep networks,' forming the basis of deep learning.

Click to watch →

The Most Important Algorithm in Machine Learning

The Most Important Algorithm in Machine Learning

939.1K

Feedforward neural networks are essentially sequences of multiplications, summations, and nonlinear activation functions between layers. Each atomic computation (multiplication, addition, activation) is differentiable, allowing the construction of a complete computational graph. By running the backward pass on this graph, we can find how each connection weight between neurons influences the loss function. Since neural networks with sufficient neurons can theoretically approximate any function, we can create large enough sequences of these building blocks to solve complex problems like image classification and text generation. The same backpropagation algorithm used for curve fitting applies universally to neural networks.

Click to watch →

Complete Deep Learning - Part 2 | Hindi

Complete Deep Learning - Part 2 | Hindi

717

Convolutional Neural Networks (CNNs) are specialized neural networks designed for processing structured grid data like images, using convolutional layers with filters (kernels) to detect features such as edges and patterns, followed by pooling layers for dimensionality reduction and fully connected layers for classification. The backpropagation algorithm in CNNs uses the chain rule to compute gradients of the loss function with respect to all trainable parameters (weights and biases) by propagating errors backward through the network, enabling the network to learn optimal parameters through gradient descent optimization.

Click to watch →

Deep Dive

The role of loss functions, optimization, and how gradient descent updates model parameters to minimize error.

Gradient Descent e Loss Function | #02 Matemática para Machine Learning

Gradient Descent e Loss Function | #02 Matemática para Machine Learning

865

In machine learning, loss functions measure model error by quantifying the difference between predicted and actual values, with Mean Squared Error (MSE) being preferred over simple error summation because it eliminates cancellation of positive and negative errors and provides a smooth, differentiable function for optimization. Gradient descent is an iterative optimization algorithm that minimizes the loss function by updating model parameters (weights and bias) in the direction opposite to the gradient (derivative) of the loss function, using the update rule: θ_new = θ_old - learning_rate × gradient(θ), where the learning rate controls the step size. This process continues until the model converges to a minimum error, enabling the model to learn optimal parameters that best fit the training data.

Click to watch →

How Machines Teach Themselves

How Machines Teach Themselves

132

Every machine learning model contains parameters (m, c in linear regression; W, B in logistic regression; theta in neural networks) that constitute the loss function. The goal of any ML algorithm is to minimize this loss function to find optimal parameters. Gradient descent is an iterative optimization algorithm that moves from high loss (top of mountain) toward minimum loss (bottom of valley). The update rule is: new_parameter = old_parameter - (learning_rate × gradient). This algorithm is model-agnostic, working for any loss function regardless of parameter count.

Click to watch →

Loss Function | Machine Learning

Loss Function | Machine Learning

3.6K

A loss function calculates the difference between predicted output and actual output values. Gradient descent is an optimization algorithm that uses the loss function to minimize this difference by adjusting model parameters. The loss function quantifies how well the model's predictions match the actual data, and gradient descent iteratively reduces this error to improve model performance.

Click to watch →

Backpropagation And Gradient Descent In Neural Networks | Neural Network Tutorial | Simplilearn

Backpropagation And Gradient Descent In Neural Networks | Neural Network Tutorial | Simplilearn

68.6K

Loss functions quantify prediction error using mathematical formulas like squared error: loss = (actual - predicted)². Gradient descent optimizes weights by visualizing loss versus weight curves and following slopes toward minima. Positive slopes indicate weight increases, negative slopes indicate decreases, and zero slope signals optimal weights. This mathematical framework transforms the learning problem into an optimization task where the goal is finding the bottom of the loss curve.

Click to watch →

Машинное обучение в аналитике: нейронные сети | Вебинар Анатолия Карпова | karpov.courses

Машинное обучение в аналитике: нейронные сети | Вебинар Анатолия Карпова | karpov.courses

6.7K

The loss function quantifies prediction error, commonly using sum of squared errors for regression. For a constant model Y = b0, the loss is SSE = (y1-b0)² + (y2-b0)² + (y3-b0)². Gradient descent minimizes this function by: (1) starting with random parameters, (2) computing the gradient (derivative) of loss with respect to each parameter, (3) updating parameters by moving opposite to the gradient, and (4) repeating until convergence. The learning rate controls step size, balancing convergence speed against overshooting the minimum.

Click to watch →

Deep Dive

Basic vector calculus and linear algebra, particularly understanding how gradients represent the direction of steepest ascent/descent in high-dimensional spaces.

Direction and Slope of Steepest Ascent/Descent with the Gradient - Calculus 3

Direction and Slope of Steepest Ascent/Descent with the Gradient - Calculus 3

314

The gradient vector (∇f = (∂f/∂x, ∂f/∂y)) automatically points in the direction of steepest ascent, and its magnitude equals the slope of steepest ascent; to find the unit directional vector for steepest ascent, divide the gradient by its magnitude, and for steepest descent, multiply the gradient by -1.

Click to watch →

Vector Calculus Complete Animated Course for DUMMIES

Vector Calculus Complete Animated Course for DUMMIES

114.1K

The gradient vector always points in the direction of greatest increase or decrease of a function at any given point. For a hill represented by a two-variable function, the gradient at each point indicates the path of steepest ascent. When the gradient is zero, it indicates a flat region (saddle point or maximum/minimum). The gradient can be calculated for explicit functions and converted to implicit form to obtain normal vectors of surfaces in three dimensions.

Click to watch →

Lecture 03: Vector Calculus (P)Review (CMU 15-462/662)

Lecture 03: Vector Calculus (P)Review (CMU 15-462/662)

30.4K

The gradient points in the direction of steepest ascent of the function. Moving in the gradient direction increases the function value most quickly, while moving in the opposite direction decreases it most quickly. Moving orthogonal to the gradient doesn't change the function value. Near a point x0, a function f can be approximated by f(x) ≈ f(x0) + ∇f(x0)·(x - x0), which is the first-order Taylor approximation. For the function f(u) = u^T v (the dot product of two vectors u and v), the gradient with respect to u is simply v. To differentiate expressions involving matrices, one strategy is to write everything out in ordinary coordinates and take partial derivatives. Common matrix derivative patterns include: gradient of x^T y with respect to x is y; gradient of x^T A x with respect to x is (A + A^T)x (if A is symmetric, this is 2Ax); gradient of x^T x with respect to x is 2x.

Click to watch →

Gradient descent, Newton's method

Gradient descent, Newton's method

43.4K

The gradient of a multivariate function f(x,y) is a vector containing all first partial derivatives (∂f/∂x, ∂f/∂y). This vector points in the direction of greatest increase of the function at any given point. For maximization problems, the gradient itself serves as the ascent direction. For minimization problems, the negative gradient (-∇f) provides the descent direction, pointing toward the steepest decrease. This principle generalizes to functions with more than two variables.

Click to watch →

Directions of Steepest Ascent & Descent

Directions of Steepest Ascent & Descent

17.5K

The gradient vector at a point gives the direction of steepest ascent; the unit vector in this direction is the gradient divided by its magnitude, while the direction of steepest descent is the negative of this unit vector. A vector pointing in the direction of no change in the function is orthogonal to the gradient vector, meaning their dot product equals zero.

Click to watch →

Deep Dive

Standard machine learning evaluation concepts, including training vs. testing distributions, generalization, and decision boundaries.

Lecture 2 "Supervised Learning Setup Continued" -Cornell CS4780 SP17

Lecture 2 "Supervised Learning Setup Continued" -Cornell CS4780 SP17

72.1K

Generalization is the ability of a learned model to perform well on new, unseen data drawn from the same underlying distribution. While we can measure training loss, what we truly want is low expected loss on any new data point. A famous example: the US Army trained a military/civilian vehicle classifier achieving zero error on training data but 50% error when deployed. The issue was distribution shift—training data was collected in bright daylight at the base, while deployment occurred in different lighting conditions. To estimate generalization performance, we split data into training and test sets. The training set is used to learn the model, while the test set is kept hidden and used only for evaluation after learning. This provides an unbiased estimate of real-world performance. The memorization algorithm would show zero training loss but high test loss, revealing its failure to generalize.

Click to watch →

Applying Machine Learning | ML-005 Lecture 10 | Stanford University | Andrew Ng

Applying Machine Learning | ML-005 Lecture 10 | Stanford University | Andrew Ng

3.4K

Effective machine learning practitioners systematically diagnose problems rather than randomly trying solutions. Common approaches like collecting more training data or adding features often fail because they don't address underlying issues. The standard evaluation procedure splits data into training (70%) and test (30%) sets. Training set learns parameters θ, while test set computes J_test(θ) using average squared error. For classification, misclassification error measures fraction of mislabeled examples. Crucially, training set error is not a reliable predictor of generalization performance because parameters are fit to the training data itself. This fundamental limitation necessitates separate test sets for unbiased evaluation of learned hypotheses.

Click to watch →

Bias Variance Trade-off Easily Explained | Machine Learning Basics

Bias Variance Trade-off Easily Explained | Machine Learning Basics

8.9K

Machine learning requires dividing data into training and testing sets. Training data teaches the model, while testing data (never seen before) evaluates generalization. Using the student exam analogy: if students only memorize assignment questions without understanding concepts, they fail on new exam questions. Similarly, models must learn underlying patterns rather than memorizing training examples to perform well on unseen data.

Click to watch →

ECE595ML Lecture 24-1 Probably Approximately Correct

ECE595ML Lecture 24-1 Probably Approximately Correct

1.7K

Generalization is the ability to predict outcomes on new data based on training data. Training and testing errors converge as the number of samples increases because they both evaluate the same underlying probability of error. The key insight is that learning is only feasible when training and testing samples are drawn from the same distribution P(x); if they come from different distributions (e.g., training on citrus fruits but testing only on oranges), generalization fails due to distribution mismatch.

Click to watch →

Introduction - Machine Learning # 1

Introduction - Machine Learning # 1

83.2K

To evaluate model generalization, split data into training (80%) and test (20%) sets. Train on training data, evaluate on test data as if they were new cases. If training error is low but test error is high, the model is overfitting. Cross-validation decomposes the training set into multiple subsets, trains on different combinations, and validates on remaining parts to select optimal hyperparameters before final testing.

Click to watch →

Fooling Networks

Why Examples Fool

Linear Model Link

Attack by Sign

Adversarial Spaces

Clever Hans Effect

Transfer Attacks

Physical And Human

Defense Hard

Path Forward

Fooling Networks

7:34

Playing Section

1
Gradient ascent on input creates unrecognizable images that networks classify with high confidence.
2
This reveals vulnerabilities common to all machine learning algorithms, not just deep neural networks.

Deep Dive

Mathematical formulation of specific adversarial attack methodologies, such as the Fast Gradient Sign Method (FGSM), Projected Gradient Descent (PGD), and Carlini-Wagner attacks.

SECOMP 2025 - Segurança em Aprendizado de Máquina: desafios e oportunidades - Erikson Aguiar

SECOMP 2025 - Segurança em Aprendizado de Máquina: desafios e oportunidades - Erikson Aguiar

415

The speaker defines the security machine learning pipeline including model poisoning during training, evasion attacks during testing, and backdoor attacks with specific triggers. The mathematical formulation involves minimizing a loss function balancing keeping inputs similar to originals while maximizing model loss. The speaker distinguishes between white box attacks (full model knowledge) and black box attacks (limited knowledge). Various attack methods are described: FGSM (Fast Gradient Sign Method), Projected Gradient Descent (PGD), One Pixel Attack, and Universal Perturbations. The Carlini-Wagner attack is highlighted as one of the most effective methods. Tools like SECML are available for experimenting with attacks. The speaker notes that attacks can reduce model accuracy significantly (from 80% to 0%) while making minimal changes to inputs, demonstrating the real-world dangers of adversarial attacks.

Click to watch →

Adversarial Examples for Deep Neural Networks

Adversarial Examples for Deep Neural Networks

12.8K

White box attacks exploit gradient information to generate adversarial examples. Projected Gradient Descent (PGD) iteratively adds gradient ascent steps to the input, then projects back onto a small-perturbation constraint (ε-ball under L∞ norm), creating progressively adversarial images. The Fast Gradient Sign Method (FGSM) achieves the same in one step: X* = X + ε × sign(∇ₓL), making it computationally efficient but weaker. The Carlini-Wagner attack finds the smallest imperceptible perturbation by minimizing the perturbation norm plus a penalty for classification errors, using differentiable logit-based constraints. All three methods require gradient access and can be adapted for targeted or untargeted attacks. These techniques demonstrate how small, carefully crafted perturbations can systematically fool neural networks.

Click to watch →

Defense Against Adversarial Attacks

Defense Against Adversarial Attacks

27.2K

The Fast Gradient Sign Method (FGSM) exploits the mathematical structure of classification boundaries. In a two-dimensional space with two classes, classifiers create decision boundaries (like straight lines) separating different classes. FGSM transforms data points toward the decision boundary in the direction of the opposite class using the gradient vector as the transformation direction. The formula X_adversarial = X_original + ε × sign(∇ₓJ(X, y)) shows how adversarial examples are created by adding noise in the gradient direction. This same gradient descent mechanism used to train models can be repurposed by attackers to create deceptive inputs.

Click to watch →

CAP6412 21Spring-Introduction Lecture -1

CAP6412 21Spring-Introduction Lecture -1

1.6K

Fast Gradient Sign Method (FGSM) is a one-shot, untargeted, white box attack that computes the gradient of loss with respect to the input image and adds epsilon times the sign of this gradient. Iterative FGSM applies this process multiple times with clipping to keep values within valid ranges. Momentum Iterative FGSM incorporates momentum from previous iterations, similar to training neural networks, to achieve better convergence. Projected Gradient Descent (PGD) is a multi-shot, targeted, white box attack that generalizes iterative gradient methods. It takes gradient steps and projects perturbations back into constraint balls defined by specific norms. For L2 norm projection, vectors are translated, scaled, projected onto unit balls, then transformed back. PGD is considered one of the most effective adversarial attack methods.

Click to watch →

Guillermo Ortiz-Jimenez (EPFL-CIS) - Catastrophic overfitting is a bug but also a feature

Guillermo Ortiz-Jimenez (EPFL-CIS) - Catastrophic overfitting is a bug but also a feature

160

Projected Gradient Descent (PGD) performs multiple gradient ascent steps while projecting back to stay within ε-balls, making it highly effective at finding adversarial examples but computationally expensive. The Fast Gradient Sign Method (FGSM) approximates PGD with a single step, computing the gradient sign and adding ε×sign(∇xL). FGSM succeeds because for small ε, the first-order linear approximation of the loss function is accurate. Despite being computationally efficient, FGSM suffers from catastrophic overfitting during adversarial training, where models appear robust against FGSM but fail against stronger attacks like PGD50.

Click to watch →

Deep Dive

The connection to Generative Adversarial Networks (GANs), which leverage competing neural networks for generative modeling.

What Are GANs? | Generative Adversarial Networks Explained | Deep Learning With Python | Edureka

What Are GANs? | Generative Adversarial Networks Explained | Deep Learning With Python | Edureka

143.9K

Generative Adversarial Networks (GANs) are deep learning-based generative models used for unsupervised learning. They consist of two competing neural networks that compete with each other to create variations in data. GANs were first described in a 2014 paper by Ian Goodfellow, and a standardized, more stable model theory called DCGAN (Deep Convolutional GAN) was proposed by Alec Radford in 2016. Most modern GAN implementations use this DCGAN architecture. The core idea is that through competition between these networks, the generator learns to produce increasingly realistic data samples.

Click to watch →

Generative adversarial networks | GAN

Generative adversarial networks | GAN

239

Generative Adversarial Networks (GANs) are neural networks that generate new data samples by having two competing networks: a generator that creates synthetic data and a discriminator that classifies samples as real or fake. These networks play a minimax game where the generator aims to fool the discriminator, and the discriminator aims to correctly identify real versus generated samples. GANs are widely applied in medical imaging, self-driving cars, image restoration, enhancement, segmentation, translation, face swapping, and deepfake creation. The training process involves the generator minimizing loss when successfully deceiving the discriminator, indicating both networks have improved through adversarial competition.

Click to watch →

Neural Networks One Shot | Expected Questions + PYQs | CBSE Class 12 AI 843 | 2026

Neural Networks One Shot | Expected Questions + PYQs | CBSE Class 12 AI 843 | 2026

910

Generative Adversarial Networks (GANs) consist of two competing neural networks: a generator and a discriminator. The generator creates synthetic data samples (such as images), while the discriminator acts as a classifier that tries to distinguish between real and generated samples. The two networks play a minimax game: the generator tries to create increasingly realistic samples to fool the discriminator, while the discriminator tries to become better at identifying fake samples. Through this adversarial training process, both networks improve iteratively, with the generator eventually producing high-quality synthetic data that closely resembles real data distributions.

Click to watch →

DLVC - 41 - Generative Adversarial Networks - Part 3 / X

DLVC - 41 - Generative Adversarial Networks - Part 3 / X

117

GANs consist of two neural networks competing against each other. The generator is a neural network (typically CNN) that takes random noise as input and generates fake images. The discriminator is another neural network that analyzes images and classifies them as real or fake. The generator aims to create images indistinguishable from real data, while the discriminator tries to correctly identify whether an image is real or generated. This adversarial competition drives both networks to improve.

Click to watch →

What Are GANs? | Generative Adversarial Networks Tutorial | Deep Learning | Generative Models

What Are GANs? | Generative Adversarial Networks Tutorial | Deep Learning | Generative Models

136

Generative Adversarial Networks (GANs) are a type of generative model in deep learning that consist of two neural networks—the generator and the discriminator—competing in a zero-sum game where the generator creates synthetic data (like images) and the discriminator tries to distinguish between real and fake data, with the goal of generating increasingly realistic outputs through adversarial training.

Click to watch →

Deep Dive

Advanced defense strategies and the concept of certifiable robustness, such as randomized smoothing and formal verification techniques.

Generalizable Adversarial Robustness to Unforeseen Attacks - Soheil Feizi

Generalizable Adversarial Robustness to Unforeseen Attacks - Soheil Feizi

1.7K

This section covers certifiable robustness and the mathematical boundaries of randomized smoothing. Certifiable robustness guarantees identical classifier outputs for any X' within a threat model. Provable defenses use varying amounts of network information: input-output relationships only, gradients, or higher-order curvature. Randomized smoothing is the most scalable provable defense against LP attacks, adding Gaussian noise and averaging outputs to smooth decision boundaries. The robustness certificate states no adversarial example exists within radius ρ = Φ⁻¹(p₁) × σ. An alternative proof shows the smoothed function is Lipschitz continuous with constant 1/σ. However, for symmetric iid smoothing distributions against LP attacks, there exists an upper bound on robustness that depends on dimension D. For p > 2, this bound decreases to zero as dimension increases—a curse of dimensionality. This reveals fundamental limitations of smoothing-based defenses in high-dimensional settings.

Click to watch →

SaTML 2023 - Zico Kolter - Robustness in Machine Learning: A Five-Year Retrospective

SaTML 2023 - Zico Kolter - Robustness in Machine Learning: A Five-Year Retrospective

3.7K

Randomized smoothing provides certified robustness by: (1) randomly sampling Gaussian noise to perturb inputs, (2) using majority vote over classifications of these perturbations to classify the original example. The key insight is that the best an adversary could do is make the classifier linear, and for linear classifiers, exact robustness bounds can be derived. This provides guarantees for any classifier as a black box, independent of its architecture.

Click to watch →

Randomized smoothing for certified robustness

Randomized smoothing for certified robustness

5.1K

Randomized smoothing provides a method to certify robustness of neural networks by convolving the classifier output with a Gaussian distribution; the certified robustness radius at a point is given by ε = σ/2 × [Φ⁻¹(PA) - Φ⁻¹(PB)], where PA and PB are the probabilities of the most likely and second most likely classes respectively, and Φ⁻¹ is the inverse cumulative distribution function of the standard Gaussian. This approach leverages the observation that when the predicted class probability is high (PA close to 1), the local Lipschitz constant becomes very small, enabling large robustness radii.

Click to watch →

Li Xiong (Emory University), Trustworthy ML with Both Differential Privacy and Certified Robustness

Li Xiong (Emory University), Trustworthy ML with Both Differential Privacy and Certified Robustness

551

Certified robustness provides provable guarantees that models correctly classify inputs within a certain radius. Randomized smoothing achieves this by transforming a base classifier into a smooth classifier that aggregates predictions from multiple Gaussian-corrupted versions of the input. This approach can certify robustness against adversarial examples by proving that the majority vote prediction remains consistent within a certified radius, offering principled defense beyond empirical methods.

Click to watch →

Certified Robustness: Fundamentals and Challenges. Aleksandr Petiushko, Nuro, USA

Certified Robustness: Fundamentals and Challenges. Aleksandr Petiushko, Nuro, USA

721

This comprehensive section covers the complete landscape of certified robustness in machine learning. It begins with foundational concepts: robustness defined as consistent outputs for similar inputs, divided into generalization robustness (handling distributional shifts) and noise robustness (resisting input corruption). The field comprises three interconnected pillars: attack generation, defense mechanisms, and certification/verification. Two core approaches exist: deterministic (finding perturbation sets preserving output) and probabilistic (computing retention probabilities). Certified robustness provides lower bounds on true robust accuracy, unlike empirical approaches offering only upper bounds. The section details randomized smoothing—a technique addressing adversarial vulnerability by adding Gaussian noise during inference and taking majority votes over multiple noisy versions. The key theorem states that if the base classifier is robust to Gaussian noise, the smoothed classifier achieves certified robustness within radius r_cap = σ/2 × Φ⁻¹(P_A - P_B). Advanced techniques include black box smoothing using external denoisers, training custom denoisers with balanced losses, and diffusion models achieving 14-30% improvement. For feature vector outputs, certified robustness depends on Lipschitz constants and distance to decision boundaries. The section reveals fundamental limitations: in high-dimensional spaces (millions of pixels), certified radii for p > 2 decrease proportionally to 1/√D, making traditional certification practically useless for computer vision.

Click to watch →

Deep Dive

Real-world security implications and threat modeling in physical-world scenarios, such as fooling autonomous vehicle vision systems or facial recognition systems.

Adversarial Attacks on AI Systems

Adversarial Attacks on AI Systems

996

Adversarial examples extend beyond digital domains to physical reality. Printed adversarial examples photographed with smartphones still fool models. Adversarial patches are specially designed stickers that cause misclassification while appearing innocuous. These physical attacks demonstrate real-world vulnerabilities for applications like autonomous vehicles, where attackers could create stop signs appearing as yield signs. The transferability of these attacks across different models and transformations (printing, lighting) reveals fundamental security gaps in deploying ML systems in safety-critical environments.

Click to watch →

É assim que as Redes Neurais são HACKEADAS!

É assim que as Redes Neurais são HACKEADAS!

213.9K

Adversarial attacks on object recognition networks are dangerous because autonomous vehicles use these networks to identify objects on the road. If misclassified, vehicles could accelerate instead of stopping. Researchers demonstrated this by adding adhesive tape to a traffic sign, causing the network to misclassify a 'Pari' sign as a '45 mph speed limit' sign. This shows how anyone could potentially attack autonomous vehicles.

Click to watch →

Paper review - Threat of Adversarial Attacks on Deep Learning in Computer Vision: A Survey | AISC

Paper review - Threat of Adversarial Attacks on Deep Learning in Computer Vision: A Survey | AISC

621

Adversarial attacks extend from laboratory settings to real-world applications with significant safety implications. Facial attribute manipulation attacks can flip binary classifications like lipstick presence or gender with imperceptible changes. Road sign attacks achieve 100% success rates by altering signs through stickers, graffiti, or color changes, potentially causing autonomous vehicles to violate traffic laws. 3D printed adversarial objects demonstrate that physical objects can be designed to exploit known computer vision perception characteristics, being classified with full confidence as false identities despite appearing normal to humans. These examples reveal critical vulnerabilities in autonomous driving systems that depend on accurate environmental perception.

Click to watch →

Физические атаки на нейросети: как футболки и наклейки обманывают ИИ

Физические атаки на нейросети: как футболки и наклейки обманывают ИИ

1K

Physical adversarial attacks differ from digital attacks because they must work in the real physical world. Challenges include not knowing the target system's architecture, the need for physical printing methods, and the requirement to work under various viewing angles and lighting conditions. Real-world examples include stickers on road signs that fool autonomous vehicles, colored tape on camera lenses causing misidentification, and specially designed masks that evade facial recognition. The 'Adversarial Fashion' project demonstrates that shirts with specific patterns can make wearers invisible to facial recognition systems, successfully evading detection in approximately 57% of cases.

Click to watch →

Guest Lecture: Adversarial Machine Learning (UMich EECS 498-007)

Guest Lecture: Adversarial Machine Learning (UMich EECS 498-007)

442

Adversarial attacks extend beyond digital manipulations to physical modifications of real-world objects. Researchers demonstrated that stop signs with strategically placed stickers can be misclassified by autonomous vehicle systems as different signs (like 45 mph speed limit signs) with high confidence. These physical attacks were successfully tested in real-world conditions and even displayed at the London Museum. DARPA and other agencies have funded extensive research on this problem due to its serious implications for autonomous systems safety. The work shows that attackers don't need sophisticated digital tools—simple stickers or patches can compromise machine vision systems used in critical infrastructure.

Click to watch →

Fooling Networks

Why Examples Fool

Linear Model Link

Attack by Sign

Adversarial Spaces

Clever Hans Effect

Transfer Attacks

Physical And Human

Defense Hard

Path Forward

Fooling Networks

7:34

Playing Section

1
Gradient ascent on input creates unrecognizable images that networks classify with high confidence.
2
This reveals vulnerabilities common to all machine learning algorithms, not just deep neural networks.

224.7K views2.5K1:21:45@stanfordengineeringOriginal Release: 2017-08-11

Adversarial examples are inputs that have been carefully modified to cause machine learning models to make confident but incorrect predictions, exploiting the fact that deep neural networks behave approximately linearly in high-dimensional input spaces, allowing attackers to find directions in input space that cause misclassification with minimal perceptible changes to human observers; these vulnerabilities exist across virtually all machine learning algorithms, not just deep learning, and can be mitigated through adversarial training where models are trained on both clean and adversarial examples to improve robustness.

[00:00:08]- Okay, sounds like it is.

[00:00:10]I'll be telling you about adversarial examples and adversarial training today.

[00:00:15]Thank you.

[00:00:18]As an overview, I will start off by telling you what adversarial examples are, and then I'll explain why they happen, why it's possible for them to exist.

[00:00:28]I'll talk a little bit about how adversarial examples pose real world security threats, that they can actually be used to compromise systems built on machine learning.

[00:00:38]I'll tell you what the defenses are so far, but mostly defenses are an open research problem that I hope some of you will move on to tackle.

[00:00:47]And then finally I'll tell you how to use adversarial examples to improve other machine learning algorithms even if you want to build a machine learning algorithm that won't face a real world adversary.

[00:01:00]Looking at the big picture and the context for this lecture, I think most of you are probably here because you've heard how incredibly powerful and successful machine learning is, that very many different tasks that could not be solved with software before are now solvable thanks to deep learning and convolutional networks and gradient descent.

[00:01:23]All of these technologies that are working really well.

[00:01:27]Until just a few years ago, these technologies didn't really work.

[00:01:30]In about 2013, we started to see that deep learning achieved human level performance at a lot of different tasks.

[00:01:39]We saw that convolutional nets could recognize objects and images and score about the same as people in those benchmarks, with the caveat that part of the reason that algorithms score as well as people is that people can't tell Alaskan Huskies from Siberian Huskies very well, but modulo the strangeness of the benchmarks deep learning caught up to about human level performance for object recognition in about 2013.

[00:02:05]That same year, we also saw that object recognition applied to human faces caught up to about human level.

[00:02:12]That suddenly we had computers that could recognize faces about as well as you or I could recognize faces of strangers.

[00:02:21]You can recognize the faces of your friends and family better than a computer, but when you're dealing with people that you haven't had a lot of experience with the computer caught up to us in about 2013.

[00:02:34]We also saw that computers caught up to humans for reading type written fonts in photos in about 2013.

[00:02:42]It even got the point that we could no longer use CAPTCHAs to tell whether a user of a webpage is human or not because the convolutional network is better at reading obfuscated text than a human is.

[00:02:56]So with this context today of deep learning working really well especially for computer vision it's a little bit unusual to think about the computer making a mistake.

[00:03:07]Before about 2013, nobody was ever surprised if the computer made a mistake.

[00:03:12]That was the rule not the exception, and so today's topic which is all about unusual mistakes that deep learning algorithms make this topic wasn't really a serious avenue of study until the algorithms started to work well most of the time, and now people study the way that they break now that that's actually the exception rather than the rule.

[00:03:36]An adversarial example is an example that has been carefully computed to be misclassified.

[00:03:43]In a lot of cases we're able to make the new image indistinguishable to a human observer from the original image.

[00:03:50]Here, I show you one where we start with a panda.

[00:03:52]On the left this is a panda that has not been modified in any way, and the convolutional network trained on the image in that dataset is able to recognize it as being a panda.

[00:04:03]One interesting thing is that the model doesn't have a whole lot of confidence in that decision.

[00:04:08]It assigns about 60% probability to this image being a panda.

[00:04:13]If we then compute exactly the way that we could modify the image to cause the convolutional network to make a mistake we find that the optimal direction to move all the pixels is given by this image in the middle.

[00:04:27]To a human it looks a lot like noise.

[00:04:29]It's not actually noise.

[00:04:31]It's carefully computed as a function of the parameters of the network.

[00:04:34]There's actually a lot of structure there.

[00:04:36]If we multiply that image of the structured attack by a very small coefficient and add it to the original panda we get an image that a human can't tell from the original panda.

[00:04:49]In fact, on this slide there is no difference between the panda on the left and the panda on the right.

[00:04:56]When we present the image to convolutional network we use 32-bit floating point values.

[00:05:01]The monitor here can only display eight bits of color resolution, and we have made a change that's just barely too small to affect the smallest of those eight bits, but it effects the other 24 of the 32-bit floating point representation, and that little tiny change is enough to fool the convolutional network into recognizing this image of a panda as being a gibbon.

[00:05:26]Another interesting thing is that it doesn't just change the class.

[00:05:29]It's not that we just barely found the decision boundary and just barely stepped across it.

[00:05:34]The convolutional network actually has much more confidence in its incorrect prediction, that the image on the right is a gibbon, than it had for the original being a panda.

[00:05:45]On the right, it believes that the image is a gibbon with 99.9% probability, so before it thought that there was about 1/3 chance that it was something other than a panda, and now it's about as certain as it can possibly be that it's a gibbon.

[00:06:02]As a little bit of history, people have studied ways of computing attacks to fool different machine learning models since at least about 2004, and maybe earlier.

[00:06:13]For a long time this was done in the context of fooling spam detectors.

[00:06:17]In about 2013, Battista Biggio found that you could fool neural networks in this way, and around the same time my colleague, Christian Szegedy, found that you could make this kind of attack against deep neural networks just by using an optimization algorithm to search on the input of the image.

[00:06:36]A lot of what I'll be telling you about today is my own follow-up work on this topic, but I've spent a lot of my career over the past few years understanding why these attacks are possible and why it's so easy to fool these convolutional networks.

[00:06:52]When my colleague, Christian, first discovered this phenomenon independently from Battista Biggio but around the same time, he found that it was actually a result of a visualization he was trying to make.

[00:07:08]He wasn't studying security.

[00:07:10]He wasn't studying how to fool a neural network.

[00:07:12]Instead, he had a convolutional network that could recognize objects very well, and he wants to understand how it worked, so he thought that maybe he could take an image of a scene, for example a picture of a ship, and he could gradually transform that image into something that the network would recognize as being an airplane.

[00:07:33]Over the course of that transformation, he could see how the features of the input change.

[00:07:38]You might expect that maybe the background 167 00:07:34,360 --> 00:07:37,692 would turn blue to look like the sky behind an airplane, or you might expect that the ship would grow wings to look more like an airplane.

[00:07:48]You could conclude from that that the convolution uses the blue sky or uses the wings to recognize airplanes.

[00:07:56]That's actually not really what happened at all.

[00:07:59]Each of these panels here shows an animation that you read left to right, top to bottom.

[00:08:03]Each panel is another step of gradient ascent on the log probability that the input is an airplane according to a convolutional net model, and then we follow the gradient on the input to the image.

[00:08:18]You're probably used to following the gradient on the parameters of a model.

[00:08:22]You can use the back propagation algorithm to compute the gradient on the input image using exactly the same procedure that you would use to compute the gradient on the parameters.

[00:08:31]In this animation of the ship in the upper left, we see five panels that all look basically the same.

[00:08:37]Gradient descent doesn't seem to have moved the image at all, but by the last panel the network is completely confident that this is an airplane.

[00:08:45]When you first code up this kind of experiment, especially if you don't know what's going to happen, it feels a little bit like you have a bug in your script and you're just displaying the same image over and over again.

[00:08:54]The first time I did it, I couldn't believe it was happening, and I had to open up the images in NumPy, and take the difference of them, and make sure that there was actually a non-zero difference in there, but there is.

[00:09:07]I show several different animations here of a ship, a car, a cat, and a truck.

[00:09:13]The only one where I actually see any change at all is the image of the cat.

[00:09:18]The color of the cat's face changes a little bit, and maybe it becomes a little bit more like the color of a metal airplane.

[00:09:25]Other than that, I don't see any changes in any of these animations, and I don't see anything very suggestive of an airplane.

[00:09:33]So gradient descent, rather than turning the input into an example of an airplane, has found an image that fools the network into thinking that the input is an airplane.

[00:09:45]And if we were malicious attackers we didn't even have to work very hard to figure out how to fool the network.

[00:09:51]We just asked the network to give us an image of an airplane, and it gave us something that fools it into thinking that the input is an airplane.

[00:10:00]When Christian first published this work, a lot of articles came out with titles like, The Flaw Looking At Every Deep Neural Network, or Deep Learning has Deep Flaws.

[00:10:10]It's important to remember that these vulnerabilities apply to essentially every machine learning algorithm that we've studied so far.

[00:10:18]Some of them like RBF networks and partisan density estimators are able to resist this effect somewhat, but even very simple machine learning algorithms are highly vulnerable to adversarial examples.

[00:10:32]In this image, I show an animation of what happens when we attack a linear model, so it's not a deep algorithm at all.

[00:10:38]It's just a shallow softmax model.

[00:10:41]You multiply by a matrix, you add a vector of bias terms, you apply the softmax function, and you've got your probability distribution over the 10 MNIST classes.

[00:10:51]At the upper left, I start with an image of a nine, and then as we move left to right, top to bottom, I gradually transform it to be a zero.

[00:11:00]Where I've drawn the yellow box, the model assigns high probability to it being a zero.

[00:11:05]I forget exactly what my threshold was for high probability, but I think it was around 0.9 or so.

[00:11:11]Then as we move to the second row, I transform it into a one, and the second yellow box indicates where we've successfully fooled the model into thinking it's a one with high probability.

[00:11:21]And then as you read the rest of the yellow boxes left to right, top to bottom, we go through the twos, threes, fours, and so on, until finally at the lower right we have a nine that has a yellow box around it, and it actually looks like a nine, but in this case the only reason it actually looks like a nine is that we started the whole process with a nine.

[00:11:39]We successfully swept through all 10 classes of MNIST without substantially changing the image of the digit in any way that would interfere with human recognition.

[00:11:50]This linear model was actually extremely easy to fool.

[00:11:55]Besides linear models, we've also seen that we can fool many different kinds of linear models including logistic regression and SVMs.

[00:12:04]We've also found that we can fool decision trees, and to a lesser extent, nearest neighbors classifiers.

[00:12:13]We wanted to explain exactly why this happens.

[00:12:16]Back in about 2014, after we'd published the original paper where we'd said that these problems exist, we were trying to figure out why they happen.

[00:12:25]When we wrote our first paper, we thought that basically this is a form of overfitting, that you have a very complicated deep neural network, it learns to fit the training set, its behavior on the test set is somewhat undefined, and then it makes random mistakes that an attacker can exploit.

[00:12:44]Let's walk through what that story looks like somewhat concretely.

[00:12:47]I have here a training set of three blue X's and three green O's.

[00:12:53]We want to make a classifier that can recognize X's and recognize O's.

[00:12:57]We have a very complicated classifier that can easily fit the training set, so we represent everywhere it believes X's should be with blobs of blue color, and I've drawn a blob of blue around all of the training set X's, so it correctly classifies the training set.

[00:13:13]It also has a blob of green mass showing where the O's are, and it successfully fits all of the green training set O's, but then because this is a very complicated function and it has just way more parameters than it actually needs to represent the training task, it throws little blobs of probability mass around the rest of space randomly.

[00:13:35]On the left there's a blob of green space that's kind of near the training set X's, and I've drawn a red X there to show that maybe this would be an adversarial example where we expect the classification to be X, but the model assigns O.

[00:13:48]On the right, I've shown that there's a red O where we have another adversarial example.

[00:13:53]We're very near the other O's.

[00:13:55]We might expect the model to assign this class to be an O, and yet because it's drawn blue mass there it's actually assigning it to be an X.

[00:14:04]If overfitting is really the story then each adversarial example is more or less the result of bad luck and also more or less unique.

[00:14:12]If we fit the model again or we fit a slightly different model we would expect to make different random mistakes on this points that are off the training set, but that was actually not what we found at all.

[00:14:25]We found that many different models would misclassify the same adversarial examples, and they would assign the same class to them.

[00:14:33]We also found that if we took the difference between an original example and an adversarial example then we had a direction in input space and we could add that same offset vector to any clean example, and we would almost always get an adversarial example as a result.

[00:14:52]So we started to realize that there was systematic effect going on here, not just a random effect.

[00:14:57]That led us to another idea which is that adversarial examples might actually be more like underfitting rather than overfitting.

[00:15:05]They might actually come from the model being too linear.

[00:15:09]Here I draw the same task again where we have the same manifold of O's and the same line of X's, and this time I fit a linear model to the data set rather than fitting a high capacity, non-linear model to it.

[00:15:23]We see that we get a dividing hyperplane running in between the two classes.

[00:15:29]This hyperplane doesn't really capture the true structure of the classes.

[00:15:33]The O's are clearly arranged in a C-shaped manifold.

[00:15:37]If we keep walking past the end of the O's, we've crossed the decision boundary and we've drawn a red O where even though we're very near the decision boundary and near other O's we believe that it is now an X.

[00:15:49]Similarly we can take steps that go from near X's to just over the line that are classified as O's.

[00:15:57]Another thing that's somewhat unusual about this plot is that if we look at the lower left or upper right corners these corners are very confidently classified as being X's on the lower left or O's on the upper right even though we've never seen any data over there at all.

[00:16:12]The linear model family forces the model to have very high confidence in these regions that are very far from the decision boundary.

[00:16:22]We've seen that linear models can actually assign really unusual confidence as you move very far from the decision boundary, even if there isn't any data there, but are deep neural networks actually anything like linear models?

[00:16:36]Could linear models actually explain anything about how it is that deep neural nets fail?

[00:16:41]It turns out that modern deep neural nets are actually very piecewise linear, so rather than being a single linear function they are piecewise linear with maybe not that many linear pieces.

[00:16:53]If we use rectified linear units then the mapping from the input image to the output logits is literally a piecewise linear function.

[00:17:03]By the logits I mean the un-normalized log probabilities before we apply the softmax op at the output of the model.

[00:17:11]There are other neural networks like maxout networks that are also literally piecewise linear.

[00:17:17]And then there are several that become very close to it.

[00:17:19]Before rectified linear units became popular most people used to use sigmoid units of one form or another either logistic sigmoid or hyperbolic tangent units.

[00:17:30]These sigmoidal units have to be carefully tuned, especially at initialization so that you spend most of your time near the center of the sigmoid where the sigmoid is approximately linear.

[00:17:43]Then finally, the LSTM, a kind of recurrent network that is one of the most popular recurrent networks today, uses addition from one time step to the next in order to accumulate and remember information over time.

[00:17:56]Addition is a particularly simple form of linearity, so we can see that the interaction from a very distant time step in the past and the present is highly linear within an LSTM.

[00:18:09]Now to be clear, I'm speaking about the mapping from the input of the model to the output of the model.

[00:18:14]That's what I'm saying is close to being linear or is piecewise linear with relatively few pieces.

[00:18:21]The mapping from the parameters of the network to the output of the network is non-linear because the weight matrices at each layer of the network are multiplied together.

[00:18:31]So we actually get extremely non-linear reactions between parameters and the output.

[00:18:36]That's what makes training a neural network so difficult.

[00:18:39]But the mapping from the input to the output is much more linear and predictable, and it means that optimization problems that aim to optimize the input to the model are much easier than optimization problems that aim to optimize the parameters.

[00:18:57]If we go and look for this happening in practice we can take a convolutional network and trace out a one-dimensional path through its input space.

[00:19:07]So what we're doing here is we're choosing a clean example.

[00:19:09]It's an image of a white car on a red background, and we are choosing a direction that will travel through space.

[00:19:16]We are going to have a coefficient epsilon that we multiply by this direction.

[00:19:21]When epsilon is negative 30, like at the left end of the plot, we're subtracting off a lot of this unit vector direction.

[00:19:28]When epsilon is zero, like in the middle of the plot, we're visiting the original image from the data set, and when epsilon is positive 30, like at the right end of the plot, we're adding this direction onto the input.

[00:19:42]In the panel on the left, I show you an animation where we move from epsilon equals negative 30 as up to epsilon equals positive 30.

[00:19:50]You read the animation left to right, top to bottom, and everywhere that there's a yellow box the input has correctly recognized as being a car.

[00:20:01]On the upper left, you see that it looks mostly blue.

[00:20:04]On the lower right, it's hard to tell what's going on.

[00:20:07]It's kind of reddish and so on.

[00:20:10]In the middle row, just after where the yellow boxes end you can see pretty clearly that it's a car on a red background, though the image is small on these slides.

[00:20:20]What's interesting to look at here is the logits that the model outputs.

[00:20:25]This is a deep convolutional rectified linear unit network.

[00:20:30]Because it uses rectified linear units, we know that the output is a piecewise linear function of the input to the model.

[00:20:38]The main question we're asking by making this plot is how many different pieces does this piecewise linear function have if we look at one particular cross section.

[00:20:48]You might think that maybe a deep net is going to represent some extremely wiggly complicated function with lots and lots of linear pieces no matter which cross section you look in.

[00:20:58]Or we might find that it has more or less two pieces for each function we look at.

[00:21:04]Each of the different curves on this plot is the logits for a different class.

[00:21:10]We see that out at the tails of the plot that the frog class is the most likely, and the frog class basically looks like a big v-shaped function.

[00:21:21]The logits for the frog class become very high when epsilon is negative 30 or positive 30, and they drop down and become a little bit negative when epsilon is zero.

[00:21:32]The car class, listed as automobile here, it's actually high in the middle, and the car is correctly recognized.

[00:21:42]As we sweep out to very negative epsilon, the logits for the car class do increase, but they don't increase nearly as quickly as the logits for the frog class.

[00:21:51]So, we've found a direction that's associated with the frog class and as we follow it out to a relatively large perturbation, we find that the model extrapolates linearly and begins to make a very unreasonable prediction that the frog class is extremely likely just because we've moved for a long time in this direction that was locally associated with the frog class being more likely.

[00:22:17]When we actually go and construct adversarial examples, we need to remember that we're able to get quite a large perturbation without changing the image very much as far as a human being is concerned.

[00:22:30]So here I show you a handwritten digit three, and I'm going to change it in several different ways, and all of these changes have the same L2 norm perturbation.

[00:22:40]In the top row, I'm going to change the three into a seven just by looking for the nearest seven in the training set.

[00:22:47]The difference between those two is this image that looks a little bit like the seven wrapped in some black lines.

[00:22:55]So here white pixels in the middle image in the perturbation column, the white pixels represent adding something and black pixels represent subtracting something as you move from the left column to the right column.

[00:23:08]So when we take the three and we apply this perturbation that transforms it into a seven, we can measure the L2 norm of that perturbation.

[00:23:16]And it turns out to have an L2 norm of 3.96.

[00:23:20]That gives you kind of a reference for how big these perturbations can be.

[00:23:24]In the middle row, we apply a perturbation of exactly the same size, but with the direction chosen randomly.

[00:23:30]In this case we don't actually change the class of the three at all, we just get some random noise that didn't really change the class.

[00:23:37]A human could still easily read it as being a three.

[00:23:41]And then finally at the very bottom row, we take the three and we just erase a piece of it with a perturbation of the same norm and we turn it into something that doesn't have any class at all.

[00:23:52]It's not a three, it's not a seven, it's just a defective input.

[00:23:56]All of these changes can happen with the same L2 norm perturbation.

[00:24:00]And actually a lot of the time with adversarial examples, you make perturbations that have an even larger L2 norm.

[00:24:06]What's going on is that there are several different pixels in the image, and so small changes to individual pixels can add up to relatively large vectors.

[00:24:15]For larger datasets like ImageNet, where there's even more pixels, you can make very small changes to each pixel that travel very far in vector space as measured by the L2 norm.

[00:24:26]That means that you can actually make changes that are almost imperceptible but actually move you really far and get a large dot product with the coefficients of the linear function that the model represents.

[00:24:38]It also means that when we're constructing adversarial examples, we need to make sure that the adversarial example procedure isn't able to do what happened in the top row of this slide here.

[00:24:48]So in the top row of this slide, we took the three and we actually just changed it into a seven.

[00:24:52]So when the model says that the image in the upper right is a seven, it's not a mistake.

[00:24:56]We actually just changed the input class.

[00:24:59]When we build adversarial examples, we want to make sure that we're measuring real mistakes.

[00:25:02]If we're experimenters studying how easy a network is to fool, we want to make sure that we're actually fooling it and not just changing the input class.

[00:25:11]And if we're an attacker, we actually want to make sure that we're causing misbehavior in the system.

[00:25:17]To do that, when we build adversarial examples, we use the maxnorm to constrain the perturbation.

[00:25:24]Basically this says that no pixel can change by more than some amount epsilon.

[00:25:28]So the L2 norm can get really big, but you can't concentrate all the changes for that L2 norm to erase pieces of the digit, like in the bottom row here we erased the top of a three.

[00:25:39]One very fast way to build an adversarial example is just to take the gradient of the cost that you used to train the network with respect to the input, and then take the sign of that gradient.

[00:25:51]The sign is essentially enforcing the maxnorm constraint.

[00:25:55]You're only allowed to change the input by up to epsilon at each pixel, so if you just take the sign it tells you whether you want to add epsilon or subtract epsilon in order to hurt the network.

[00:26:07]You can view this as taking the observation that the network is more or less linear, as we showed on this slide, and using that to motivate building a first order Taylor series approximation of the neural network's cost.

[00:26:21]And then subject to that Taylor series approximation, we want to maximize the cost following this maxnorm constraint.

[00:26:28]And that gives us this technique that we call the fast gradient sign method.

[00:26:32]If you want to just get your hands dirty and start making adversarial examples really quickly, or if you have an algorithm where you want to train on adversarial examples in the inner loop of learning, this method will make adversarial examples for you very, very quickly.

[00:26:45]In practice you should also use other methods, like Nicholas Carlini's attack based on multiple steps of the Adam optimizer, to make sure that you have a very strong attack that you bring out when you think you have a model that might be more powerful.

[00:26:59]A lot of the time people find that they can defeat the fast gradient sign method and think that they've built a successful defense, but then when you bring out a more powerful method that takes longer to evaluate, they find that they can't overcome the more computationally expensive attack.

[00:27:18]I've told you that adversarial examples happen because the model is very linear.

[00:27:22]And then I told you that we could use this linearity assumption to build this attack, the fast gradient sign method.

[00:27:28]This method, when applied to a regular neural network that doesn't have any special defenses, will get over a 99% attack success rate.

[00:27:38]So that seems to confirm, somewhat, this hypothesis that adversarial examples come from the model being far too linear and extrapolating in linear fashions when it shouldn't.

[00:27:48]Well we can actually go looking for some more evidence.

[00:27:51]My friend David Warde-Farley and I built these maps of the decision boundaries of neural networks.

[00:27:57]And we found that they are consistent with the linearity hypothesis.

[00:28:02]So the FGSM is that attack method that I described in the previous slide, where we take the sign of the gradient.

[00:28:08]We'd like to build a map of a two-dimensional cross section of input space and show which classes are assigned to the data at each point.

[00:28:18]In the grid on the right, each different cell, each little square within the grid, is a map of a CIFAR-10 classifier's decision boundary, with each cell corresponding to a different CIFAR-10 testing sample.

[00:28:32]On the left I show you a little legend where you can understand what each cell means.

[00:28:37]The very center of each cell corresponds to the original example from the CIFAR-10 dataset with no modification.

[00:28:45]As we move left to right in the cell, we're moving in the direction of the fast gradient sign method attack.

[00:28:50]So just the sign of the gradient.

[00:28:53]As we move up and down within the cell, we're moving in a random direction that's orthogonal to the fast gradient sign method direction.

[00:29:00]So we get to see a cross section, a 2D cross section of CIFAR-10 decision space.

[00:29:07]At each pixel within this map, we plot a color that tells us which class is assigned there.

[00:29:13]We use white pixels to indicate that the correct class was chosen, and then we used different colors to represent all of the other incorrect classes.

[00:29:21]You can see that in nearly all of the grid cells on the right, roughly the left half of the image is white.

[00:29:29]So roughly the left half of the image has been correctly classified.

[00:29:33]As we move to the right, we see that there is usually a different color on the right half.

[00:29:39]And the boundaries between these regions are approximately linear.

[00:29:43]What's going on here is that the fast gradient sign method has identified a direction where if we get a large dot product with that direction we can get an adversarial example.

[00:29:52]And from this we can see that adversarial examples live more or less in linear subspaces.

[00:29:59]When we first discovered adversarial examples, we thought that they might live in little tiny pockets.

[00:30:04]In the first paper we actually speculated that maybe they're a little bit like the rational numbers, hiding out finely tiled among the real numbers, with nearly every real number being near a rational number.

[00:30:15]We thought that because we were able to find an adversarial example corresponding to every clean example that we loaded into the network.

[00:30:22]After doing this further analysis, we found that what's happening is that every real example is near one of these linear decision boundaries where you cross over into an adversarial subspace.

[00:30:32]And once you're in that adversarial subspace, all the other points nearby are also adversarial examples that will be misclassified.

[00:30:40]This has security implications because it means you only need to get the direction right.

[00:30:46]You don't need to find an exact coordinate in space.

[00:30:48]You just need to find a direction that has a large dot product with the sign of the gradient.

[00:30:54]And once you move more or less approximately in that direction, you can fool the model.

[00:31:01]We also made another cross section where after using the left-right axis as the fast gradient sign method, we looked for a second direction that has high dot product with the gradient so we could make both axes adversarial.

[00:31:14]And in this case you see that we get linear decision boundaries.

[00:31:18]They're now oriented diagonally rather than vertically, but you can see that there's actually this two-dimensional subspace of adversarial examples that we can cross into.

[00:31:29]Finally it's important to remember that adversarial examples are not noise.

[00:31:33]You can add a lot of noise to an adversarial example and it will stay adversarial.

[00:31:37]You can add a lot of noise to a clean example and it will stay clean.

[00:31:40]Here we make random cross sections where both axes are randomly chosen directions.

[00:31:45]And you see that on CIFAR-10, most of the cells are completely white, meaning that they're correctly classified to start with, and when you add noise they stay correctly classified.

[00:31:54]We also see that the model makes some mistakes because this is the test set.

[00:31:58]And generally if a test example starts out misclassified, adding the noise doesn't change it.

[00:32:03]There are a few exceptions where, if you look in the third row, third column, noise actually can make the model misclassify the example for especially large noise values.

[00:32:14]And there's even some where, in the top row there's one example you can see where the model is misclassifying the test example to start with but then noise can change it to be correctly classified.

[00:32:26]For the most part, noise has very little effect on the classification decision compared to adversarial examples.

[00:32:33]What's going on here is that in high dimensional spaces, if you choose some reference vector and then you choose a random vector in that high dimensional space, the random vector will, on average, have zero dot product with the reference vector.

[00:32:49]So if you think about making a first order Taylor series approximation of your cost, and thinking about how your Taylor series approximation predicts that random vectors will change your cost.

[00:33:00]You see that random vectors on average have no effect on the cost.

[00:33:04]But adversarial examples are chosen to maximize it.

[00:33:10]In these plots we looked in two dimensions.

[00:33:13]More recently, Florian Tramer here at Stanford got interested in finding out just how many dimensions there are to these subspaces where the adversarial examples lie in a thick contiguous region.

[00:33:25]And we came up with an algorithm together where you actually look for several different orthogonal vectors that all have a large dot product with the gradient.

[00:33:35]By looking in several different orthogonal directions simultaneously, we can map out this kind of polytope where many different adversarial examples live.

[00:33:45]We found out that this adversarial region has on average about 25 dimensions.

[00:33:51]If you look at different examples you'll find different numbers of adversarial dimensions.

[00:33:56]But on average on MNIST we found it was about 25.

[00:33:59]So what's interesting here is the dimensionality actually tells you something about how likely you are to find an adversarial example by generating random noise.

[00:34:09]If every direction were adversarial, then any change would cause a misclassification.

[00:34:15]If most of the directions were adversarial, then random directions would end up being adversarial just by accident most of the time.

[00:34:22]And then if there was only one adversarial direction, you'd almost never find that direction just by adding random noise.

[00:34:30]When there's 25 you have a chance of doing it sometimes.

[00:34:34]Another interesting thing is that different models will often misclassify the same adversarial examples.

[00:34:39]The subspace dimensionality of the adversarial subspace relates to that transfer property.

[00:34:46]The larger the dimensionality of the subspace, the more likely it is that the subspaces for two models will intersect.

[00:34:52]So if you have two different models that have a very large adversarial subspace, you know that you can probably transfer adversarial examples from one to the other.

[00:35:01]But if the adversarial subspace is very small, then unless there's some kind of really systematic effect forcing them to share exactly the same subspace, it seems less likely that you'll be able to transfer examples just due to the subspaces randomly aligning.

[00:35:17]A lot of the time in the adversarial example research community, we refer back to the story of Clever Hans.

[00:35:25]This comes from an essay by Bob Sturm called Clever Hans, Clever Algorithms.

[00:35:30]Because Clever Hans is a pretty good metaphor for what's happening with machine learning algorithms.

[00:35:35]So Clever Hans was a horse that lived in the early 1900s.

[00:35:39]His owner trained him to do arithmetic problems.

[00:35:43]So you could ask him, "Clever Hans, "what's two plus one?"

[00:35:47]And he would answer by tapping his hoof.

[00:35:52]And after the third tap, everybody would start cheering and clapping and looking excited because he'd actually done an arithmetic problem.

[00:35:59]Well it turned out that he hadn't actually learned to do arithmetic.

[00:36:03]But it was actually pretty hard to figure out what was going on.

[00:36:06]His owner was not trying to defraud anybody, his owner actually believed he could do arithmetic.

[00:36:13]And presumably Clever Hans himself was not trying to trick anybody.

[00:36:18]But eventually a psychologist examined him and found that if he was put in a room alone without an audience, and the person asking the questions wore a mask, he couldn't figure out when to stop tapping.

[00:36:31]You'd ask him, "Clever Hans, "what's one plus one?"

[00:36:33]And he'd just [knocking] keep staring at your face, waiting for you to give him some sign that he was done tapping.

[00:36:42]So everybody in this situation was trying to do the right thing.

[00:36:46]Clever Hans was trying to do whatever it took to get the apple that his owner would give him when he answered an arithmetic problem.

[00:36:53]His owner did his best to train him correctly with real arithmetic questions and real rewards for correct answers.

[00:37:00]And what happened was that Clever Hans inadvertently focused on the wrong cue.

[00:37:07]He found this cue of people's social reactions that could reliably help him solve the problem, but then it didn't generalize to a test set where you intentionally took that cue away.

[00:37:18]It did generalize to a naturally occurring test set, where he had an audience.

[00:37:22]So that's more or less what's happening with machine learning algorithms.

[00:37:26]They've found these very linear patterns that can fit the training data, and these linear patterns even generalize to the test data.

[00:37:34]They've learned to handle any example that comes from the same distribution as their training data.

[00:37:40]But then if you shift the distribution that you test them on, if a malicious adversary actually creates examples that are intended to fool them, they're very easily fooled.

[00:37:51]In fact we find that modern machine learning algorithms are wrong almost everywhere.

[00:37:56]We tend to think of them as being correct most of the time, because when we run them on naturally occurring inputs they achieve very high accuracy percentages.

[00:38:06]But if we look instead of as the percentage of samples from an IID test set, if we look at the percentage of the space in RN that is correctly classified, we find that they misclassify almost everything and they behave reasonably only on a very thin manifold surrounding the data that we train them on.

[00:38:27]In this plot, I show you several different examples of Gaussian noise that I've run through a CIFAR-10 classifier.

[00:38:35]Everywhere that there is a pink box, the classifier thinks that there is something rather than nothing.

[00:38:40]I'll come back to what that means in a second.

[00:38:43]Everywhere that there is a yellow box, one step of the fast gradient sign method was able to persuade the model that it was looking specifically at an airplane.

[00:38:52]I chose the airplane class because it was the one with the lowest success rate.

[00:38:56]It had about a 25% success rate.

[00:38:58]That means an attacker would need four chances to get noise recognized as an airplane on this model.

[00:39:06]An interesting thing, and appropriate enough given the story of Clever Hans, is that this model found that about 70% of RN was classified as a horse.

[00:39:17]So I mentioned that this model will say that noise is something rather than nothing.

[00:39:22]And it's actually kind of important to think about how we evaluate that.

[00:39:26]If you have a softmax classifier, it has to give you a distribution over the n different classes that you train it on.

[00:39:34]So there's a few ways that you can argue that the model is telling you that there's something rather than nothing.

[00:39:39]One is you can say, if it assigns something like 90% to one particular class, that seems to be voting for that class being there.

[00:39:46]We'd much rather see it give us something like a uniform distribution saying this noise doesn't look like anything in the training set so it's equally likely to be a horse or a car.

[00:39:56]And that's not what the model does.

[00:39:58]It'll say, this is very definitely a horse.

[00:40:01]Another thing that you can do is you can replace the last layer of the model.

[00:40:05]For example, you can use a sigmoid output for each class.

[00:40:10]And then the model is actually capable of telling you that any subset of classes is present.

[00:40:14]It could actually tell you that an image is both a horse and a car.

[00:40:17]And what we would like it to do for the noise is tell us that none of the classes is present, that all of the sigmoids should have a value of less than 1/2.

[00:40:25]And 1/2 isn't even particularly a low threshold.

[00:40:29]We could reasonably expect that all of the sigmoids would be less than 0.01 for such a defective input as this.

[00:40:35]But what we find instead is that the sigmoids tend to have at least one class present just when we run Gaussian noise of sufficient norm through the model.

[00:40:48]We've also found that we can do adversarial examples for reinforcement learning.

[00:40:51]And there's a video for this.

[00:40:53]I'll upload the slides after the talk and you can follow the link.

[00:40:56]Unfortunately I wasn't able to get the WiFi to work so I can't show you the video animated.

[00:41:00]But I can describe basically what's going on from this still here.

[00:41:05]There's a game Seaquest on Atari where you can train reinforcement learning agents to play that game.

[00:41:11]And you can take the raw input pixels and you can take the fast gradient sign method or other attacks that use other norms besides the max norm, and compute perturbations that are intended to change the action that the policy would select.

[00:41:27]So the reinforcement learning policy, you can think of it as just being like a classifier that looks at a frame.

[00:41:33]And instead of categorizing the input into a particular category, it gives you a softmax distribution over actions to take.

[00:41:40]So if we just take that and say that the most likely action should have its accuracy be decreased by the adversary.

[00:41:47]Sorry, to have its probability be decreased by the adversary, you'll get these perturbations of input frames that you can then apply and cause the agent to play different actions than it would have otherwise.

[00:41:58]And using this you can make the agent play Seaquest very, very badly.

[00:42:03]It's maybe not the most interesting possible thing.

[00:42:06]What we'd really like is an environment where there are many different reward functions available for us to study.

[00:42:11]So for example, if you had a robot that was intended to cook scrambled eggs, and you had a reward function measuring how well it's cooking scrambled eggs, and you had another reward function measuring how well it's cooking chocolate cake, it would be really interesting if we could make adversarial examples that cause the robot to make a chocolate cake when the user intended for it to make scrambled eggs.

[00:42:35]That's because it's very difficult to succeed at something and it's relatively straightforward to make a system fail.

[00:42:40]So right now, adversarial examples for RL are very good at showing that we can make RL agents fail.

[00:42:45]But we haven't yet been able to hijack them and make them do a complicated task that's different from what their owner intended.

[00:42:51]Seems like it's one of the next steps in adversarial example research though.

[00:42:58]If we look at high-dimension linear models, we can actually see that a lot of this is very simple and straightforward.

[00:43:04]Here we have a logistic regression model that classifies sevens and threes.

[00:43:10]So the whole model can be described just by a weight vector and a single scalar bias term.

[00:43:16]We don't really need to see the bias term for this exercise.

[00:43:20]If you look on the left I've plotted the weights that we used to discriminate sevens and threes.

[00:43:24]The weights should look a little bit like the difference between the average seven and the average three.

[00:43:30]And then down at the bottom we've taken the sign of the weights.

[00:43:33]So the gradient for a logistic regression model is going to be proportional to the weights.

[00:43:38]And then the sign of the weights gives you essentially the sign of the gradient.

[00:43:43]So we can do the fast gradient sign method to attack this model just by looking at its weights.

[00:43:49]In the examples in the panel that's the second column from the left we can see clean examples.

[00:43:55]And then on the right we've just added or subtracted this image of the sign of the weights off of them.

[00:44:00]To you and me as human observers, the sign of the weights is just like garbage that's in the background, and we more or less filter it out.

[00:44:09]It doesn't look particularly interesting to us.

[00:44:11]It doesn't grab our attention.

[00:44:14]To the logistic regression model this image of the sign of the weights is the most salient thing that could ever appear in the image.

[00:44:22]When it's positive it looks like the world's most quintessential seven.

[00:44:26]When it's negative it looks like the world's most quintessential three.

[00:44:29]And so the model makes its decision almost entirely based on this perturbation we added to the image, rather than on the background.

[00:44:38]You could also take this same procedure, and my colleague Andrej at OpenAI showed how you can modify the image on ImageNet using this same approach, and turn this goldfish into a daisy.

[00:44:51]Because ImageNet is much higher dimensional, you don't need to use quite as large of a coefficient on the image of the weights.

[00:44:58]So we can make a more persuasive fooling attack.

[00:45:03]You can see that this same image of the weights, when applied to any different input image, will actually reliably cause a misclassification.

[00:45:12]What's going on is that there are many different classes, and it means that if you choose the weights for any particular class, it's very unlikely that a new test image will belong to that class.

[00:45:25]So on ImageNet, if we're using the weights for the daisy class, and there are 1,000 different classes, then we have about a 99.9% chance that a test image will not be a daisy.

[00:45:36]If we then go ahead and add the weights for the daisy class to that image, then we get a daisy, and because that's not the correct class, it's a misclassification.

[00:45:45]So there's a paper at CVPR this year called Universal Adversarial Perturbations that expands a lot more on this observation that we had going back in 2014.

[00:45:53]But basically these weight vectors, when applied to many different images, can cause misclassification in all of them.

[00:46:04]I've spent a lot of time telling you that these linear models are just terrible, and at some point you've probably been hoping I would give you some sort of a control experiment to convince you that there's another model that's not terrible.

[00:46:16]So it turns out that some quadratic models actually perform really well.

[00:46:21]In particular a shallow RBF network is able to resist adversarial perturbations very well.

[00:46:27]Earlier I showed you an animation where I took a nine and I turned it into a zero, one, two, and so on, without really changing its appearance at all.

[00:46:34]And I was able to fool a linear softmax regression classifier.

[00:46:39]Here I've got an RBF network where it outputs a separate probability of each class being absent or present, and that probability is given by e to the negative square of the difference between a template image and the input image.

[00:46:55]And if we actually follow the gradient of this classifier, it does actually turn the image into a zero, a one, a two, a three, and so on, and we can actually recognize those changes.

[00:47:07]The problem is, this classifier does not get very good accuracy on the training set.

[00:47:12]It's a shallow model.

[00:47:13]It's basically just a template matcher.

[00:47:15]It is literally a template matcher.

[00:47:17]And if you try to make it more sophisticated by making it deeper, it turns out that the gradient of these RBF units is zero, or very near zero, throughout most of RN.

[00:47:30]So they're extremely difficult to train, even with batch normalization and methods like that.

[00:47:36]I haven't managed to train a deep RBF network yet.

[00:47:39]But I think if somebody comes up with better hyperparameters or a new, more powerful optimization algorithm, it might be possible to solve the adversarial example problem by training a deep RBF network where the model is so nonlinear and has such wide flat areas that the adversary is not able to push the cost uphill just by making small changes to the model's input.

[00:48:05]One of the things that's the most alarming about adversarial examples is that they generalize from one dataset to another and one model to another.

[00:48:13]Here I've trained two different models on two different training sets.

[00:48:17]The training sets are tiny in both cases.

[00:48:20]It's just MNIST three versus seven classification, and this is really just for the purpose of making a slide.

[00:48:26]If you train a logistic regression model on the digits shown in the left panel, you get the weights shown on the left in the lower panel.

[00:48:35]If you train a logistic regression model on the digits shown in the upper right, you get the weights shown on the right in the lower panel.

[00:48:42]So you've got two different training sets and we learn weight vectors that look very similar to each other.

[00:48:47]That's just because machine learning algorithms generalize.

[00:48:50]You want them to learn a function that's somewhat independent of the data that you train them on.

[00:48:54]It shouldn't matter which particular training examples you choose.

[00:48:57]If you want to generalize from the training set to the test set, you've also got to expect that different training sets will give you more or less the same result.

[00:49:05]And that means that because they've learned more or less similar functions, they're vulnerable to similar adversarial examples.

[00:49:13]An adversary can compute an image that fools one and use it to fool the other.

[00:49:18]In fact we can actually go ahead and measure the transfer rate between several different machine learning techniques, not just different data sets.

[00:49:27]Nicolas Papernot and his collaborators have spent a lot of time exploring this transferability effect.

[00:49:32]And they found that for example, logistic regression makes adversarial examples that transfer to decision trees with 87.4% probability.

[00:49:43]Wherever you see dark squares in this matrix, that shows that there's a high amount of transfer.

[00:49:50]That means that it's very possible for an attacker using the model on the left to create adversarial examples for the model on the right.

[00:50:01]The procedure overall is that, suppose the attacker wants to fool a model that they don't actually have access to.

[00:50:07]They don't know the architecture that's used to train the model.

[00:50:11]They may not even know which algorithm is being used.

[00:50:13]They may not know whether they're attacking a decision tree or a deep neural net.

[00:50:17]And they also don't know the parameters of the model that they're going to attack.

[00:50:23]So what they can do is train their own model that they'll use to build the attack.

[00:50:30]There's two different ways you can train your own model.

[00:50:32]One is you can label your own training set for the same task that you want to attack.

[00:50:36]Say that somebody is using an ImageNet classifier, and for whatever reason you don't have access to ImageNet, you can take your own photos and label them, train your own object recognizer.

[00:50:46]It's going to share adversarial examples with an ImageNet model.

[00:50:50]The other thing you can do is, say that you can't afford to gather your own training set.

[00:50:55]What you can do instead is if you can get limited access to the model where you just have the ability to send inputs to the model and observe its outputs, then you can send those inputs, observe the outputs, and use those as your training set.

[00:51:09]This'll work even if the output that you get from the target model is only the class label that it chooses.

[00:51:15]A lot of people read this and assume that you need to have access to all the probability values it outputs.

[00:51:21]But even just the class labels are sufficient.

[00:51:24]So once you've used one of these two methods, either gather your own training set or observing the outputs of a target model, you can train your own model and then make adversarial examples for your model.

[00:51:36]Those adversarial examples are very likely to transfer and affect the target model.

[00:51:41]So you can then go and send those out and fool it, even if you didn't have access to it directly.

[00:51:48]We've also measured the transferability across different data sets, and for most models we find that they're kind of in an intermediate zone where different data sets will result in a transfer rate of, like, 60% to 80%.

[00:52:01]There's a few models like SVMs that are very data dependent because SVMs end up focusing on a very small subset of the training data to form their final decision boundary.

[00:52:10]But most models that we care about are somewhere in the intermediate zone.

[00:52:17]Now that's just assuming that you rely on the transfer happening naturally.

[00:52:22]You make an adversarial example and you hope that it will transfer to your target.

[00:52:26]What if you do something to stack the deck in your favor and improve the odds that you'll get your adversarial examples to transfer?

[00:52:35]Dawn Song's group at UC Berkeley studied this.

[00:52:38]They found that if they take an ensemble of different models and they use gradient descent to search for an adversarial example that will fool every member of their ensemble, then it's extremely likely that it will transfer and fool a new machine learning model.

[00:52:56]So if you have an ensemble of five models, you can get it to the point where there's essentially a 100% chance that you'll fool a sixth model out of the set of models that they compared.

[00:53:07]They looked at things like ResNets of different depths, VGG, and GoogLeNet.

[00:53:12]So in the labels for each of the different rows you can see that they made ensembles that lacked each of these different models, and then they would test it on the different target models.

[00:53:23]So like if you make an ensemble that omits GoogLeNet, you have only about a 5% chance of GoogLeNet correctly classifying the adversarial example you make for that ensemble.

[00:53:37]If you make an ensemble that omits ResNet-152, in their experiments they found that there was a 0% chance of ResNet-152 resisting that attack.

[00:53:48]That probably indicates they should have run some more adversarial examples until they found a non-zero success rate, but it does show that the attack is very powerful.

[00:53:57]And then when you go look into intentionally cause the transfer effect, you can really make it quite strong.

[00:54:05]A lot of people often ask me if the human brain is vulnerable to adversarial examples.

[00:54:10]And for this lecture I can't use copyrighted material, but there's some really hilarious things on the Internet if you go looking for, like, the fake CAPTCHA with images of Mark Hamill, you'll find something that my perception system definitely can't handle.

[00:54:29]So here's another one that's actually published with a license where I was confident I'm allowed to use it.

[00:54:35]You can look at this image of different circles here, and they appear to be intertwined spirals.

[00:54:42]But in fact they are concentric circles.

[00:54:45]The orientation of the edges of the squares is interfering with the edge detectors in your brain, making it look like the circles are spiraling.

[00:54:55]So you can think of these optical illusions as being adversarial examples in the human brain.

[00:54:59]What's interesting is that we don't seem to share many adversarial examples in common with machine learning models.

[00:55:05]Adversarial examples transfer extremely reliably between different machine learning models, especially if you use that ensemble trick that was developed at UC Berkeley.

[00:55:15]But those adversarial examples don't fool us.

[00:55:18]It tells us that we must be using a very different algorithm or model family than current convolutional networks.

[00:55:25]We don't really know what the difference is yet, but it would be very interesting to figure that out.

[00:55:30]It seems to suggest that studying adversarial examples could tell us how to significantly improve our existing machine learning models.

[00:55:37]Even if you don't care about having an adversary, we might figure out something or other about how to make machine learning algorithms deal with ambiguity and unexpected inputs more like a human does.

[00:55:52]If we actually want to go out and do attacks in practice, there's started to be a body of research on this subject.

[00:56:00]Nicolas Papernot showed that he could use the transfer effect to fool classifiers hosted by MetaMind, Amazon, and Google.

[00:56:09]So these are all just different machine learning APIs where you can upload a dataset and the API will train the model for you.

[00:56:16]And then you don't actually know, in most cases, which model is trained for you.

[00:56:21]You don't have access to its weights or anything like that.

[00:56:23]So Nicolas would train his own copy of the model using the API, and then build a model on his own personal desktop where he could fool the API hosted model.

[00:56:34]Later, Berkeley showed you could fool Clarifai in this way.

[00:56:36]Yeah?

[00:56:37]- [Man] What did you mean when you said machine having adversarial models don't generally fool us?

[00:56:41]Because I thought that was part of the point that we generally do machine-generated adversarial models where just a few pixels change.

[00:56:48]- Oh, so if we look at, for example, like this picture of the panda.

[00:56:55]To us it looks like a panda.

[00:56:56]To most machine learning models it looks like a gibbon.

[00:56:59]And so this change isn't interfering with our brains, but it fools reliably with lots of different machine learning models.

[00:57:08]I saw somebody actually took this image of the perturbation out of our paper, and they pasted it on their Facebook profile picture to see if it could interfere with Facebook recognizing them.

[00:57:20]And they said that it did.

[00:57:22]I don't think that Facebook has a gibbon tag though, so we don't know if they managed to make it think that they were a gibbon.

[00:57:34]And one of the other things that you can do that's of fairly high practical significance is you can actually fool malware detectors.

[00:57:42]Catherine Gross at the University of Saarland wrote a paper about this.

[00:57:45]And there's starting to be a few others.

[00:57:47]There's a model called MalGAN that actually uses a GAN to generate adversarial examples for malware detectors.

[00:57:54]Another thing that matters a lot if you are interested in using these attacks in the real world and defending against them in the real world is that a lot of the time you don't actually have access to the digital input to a model.

[00:58:06]If you're interested in the perception system for a self-driving car or a robot, you probably don't get to actually write to the buffer on the robot itself.

[00:58:15]You just get to show the robot objects that it can see through a camera lens.

[00:58:20]So my colleague Alexey Kurakin and Samy Bengio and I wrote a paper where we studied if we can actually fool an object recognition system running on a phone, where it perceives the world through a camera.

[00:58:33]Our methodology was really straightforward.

[00:58:35]We just printed out several pictures of adversarial examples.

[00:58:38]And we found that the object recognition system run by the camera was fooled by them.

[00:58:44]The system on the camera is actually different from the model that we used to generate the adversarial examples.

[00:58:49]So we're showing not just transfer across the changes that happen when you use the camera, we're also showing that those transfer across the model that you use.

[00:59:00]So the attacker could conceivably fool a system that's deployed in a physical agent, even if they don't have access to the model on that agent and even if they can't interface directly with the agent but just subtly modify objects that it can see in its environment.

[00:59:19]Yeah?

[00:59:20]- [Man] Why does the, for the low quality camera image noise not affect the adversarial example?

[00:59:26]Because that's what one would expect.

[00:59:28]- Yeah, so I think a lot of that comes back to the maps that I showed earlier.

[00:59:34]If you cross over the boundary into the realm of adversarial examples, they occupy a pretty wide space and they're very densely packed in there.

[00:59:43]So if you jostle around a little bit, you're not going to recover from the adversarial attack.

[00:59:48]If the camera noise, somehow or other, was aligned with the negative gradient of the cost, then the camera could take a gradient descent step downhill and rescue you from the uphill step that the adversary took.

[01:00:01]But probably the camera's taking more or less something that you could model as a random direction.

[01:00:06]Like clearly when you use the camera more than once it's going to do the same thing each time, but from the point of view of how that direction relates to the image classification problem, it's more or less a random variable that you sample once.

[01:00:22]And it seems unlikely to align exactly with the normal to this class boundary.

[01:00:33]There's a lot of different defenses that we'd like to build.

[01:00:36]And it's a little bit disappointing that I'm mostly here to tell you about attacks.

[01:00:41]I'd like to tell you how to make your systems more robust.

[01:00:44]But basically every attack we've tried has failed pretty badly.

[01:00:49]And in fact, even when people have published that they successfully defended.

[01:00:55]Well, there's been several papers on arXiv over the last several months.

[01:00:59]Nicholas Carlini at Berkeley just released a paper where he shows that 10 of those defenses are broken.

[01:01:07]So this is a really, really hard problem.

[01:01:09]You can't just make it go away by using traditional regularization techniques.

[01:01:15]Particular, generative models are not enough to solve the problem.

[01:01:19]A lot of people say, "Oh the problem that's going on here "is you don't know anything about the distribution "over the input pixels.

[01:01:25]"If you could just tell "whether the input is realistic or not "then you'd be able to resist it."

[01:01:31]It turns out that what's going on here is what matters more than getting the right distributions over the inputs x, is getting the right posterior distribution over the class of labels y given inputs x.

[01:01:42]So just using a generative model is not enough to solve the problem.

[01:01:46]I think a very carefully designed generative model could possibly do it.

[01:01:51]Here I show two different modes of a bimodal distribution, and we have two different generative models that try to capture these modes.

[01:01:58]On the left we have a mixture of two Gaussians.

[01:02:01]On the right we have a mixture of two Laplacians.

[01:02:04]You can not really tell the difference visually between the distribution they impose over x, and the difference in the likelihood they assign to the training data is negligible.

[01:02:13]But the posterior distribution they assign over classes is extremely different.

[01:02:17]On the left we get a logistic regression classifier that has very high confidence out in the tails of the distribution where there is never any training data.

[01:02:27]On the right, with the Laplacian distribution, we level off to more or less 50-50.

[01:02:33]Yeah?

[01:02:33][speaker drowned out] The issue is that it's a nonstationary distribution.

[01:02:46]So if you train it to recognize one kind of adversarial example, then it will become vulnerable to another kind that's designed to fool its detector.

[01:02:55]That's one of the category of defenses that Nicholas broke in his latest paper that he put out.

[01:03:04]So here basically the choice of exactly the family of generative model has a big effect in whether the posterior becomes deterministic or uniform, as the model extrapolates.

[01:03:17]And if we could design a really rich, deep generative model that can generate realistic ImageNet images and also correctly calculate its posterior distribution, then maybe something like this approach could work.

[01:03:31]But at the moment it's really difficult to get any of those probabilistic calculations correct.

[01:03:36]And what usually happens is, somewhere or other we make an approximation that causes the posterior distribution to extrapolate very linearly again.

[01:03:45]It's been a difficult engineering challenge to build generative models that actually capture these distributions accurately.

[01:03:55]The universal approximator theorem tells us that whatever shape we would like our classification function to have, a neural net that's big enough ought to be able to represent it.

[01:04:06]It's an open question whether we can train the neural net to have that function, but we know that we should be able to at least give the right shape.

[01:04:13]So so far we've been getting neural nets that give us these very linear decision functions, and we'd like to get something that looks a little bit more like a step function.

[01:04:21]So what if we actually just train on adversarial examples?

[01:04:25]For every input x in the training set, we also say we want you to train x plus an attack to map to the same class label as the original.

[01:04:34]It turns out that this sort of works.

[01:04:37]You can generally resist the same kind of attack that you train on.

[01:04:41]And an important consideration is making sure that you could run your attack very quickly so that you can train on lots of examples.

[01:04:48]So here the green curve at the very top, the one that doesn't really descend much at all, that's the test set error on adversarial examples if you train on clean examples only.

[01:05:00]The cyan curve that descends more or less diagonally through the middle of the plot, that's the tester on adversarial examples if you train on adversarial examples.

[01:05:10]You can see that it does actually reduce significantly.

[01:05:13]It gets down to a little bit less than 1% error.

[01:05:16]And the important thing to keep in mind here is that this is fast gradient sign method adversarial examples.

[01:05:23]It's much harder to resist iterative multi-step adversarial examples where you run an optimizer for a long time searching for a vulnerability.

[01:05:31]And another thing to keep in mind is that we're testing on the same kind of adversarial examples that we train on.

[01:05:36]It's harder to generalize from one optimization algorithm to another.

[01:05:42]By comparison, if you look at what happens on clean examples, the blue curve shows what happens on the clean test set error rate if you train only on clean examples.

[01:05:55]The red curve shows what happens if you train on both clean and adversarial examples.

[01:06:01]We see that the red curve actually drops lower than the blue curve.

[01:06:04]So on this task, training on adversarial examples actually helped us to do the original task better.

[01:06:10]This is because in the original task we were overfitting.

[01:06:12]Training on adversarial examples is good regularizer.

[01:06:15]If you're overfitting it can make you overfit less.

[01:06:18]If you're underfitting it'll just make you underfit worse.

[01:06:21]Other kinds of models besides deep neural nets don't benefit as much from adversarial training.

[01:06:27]So when we started this whole topic of study we thought that deep neural nets might be uniquely vulnerable to adversarial examples.

[01:06:33]But it turns out that actually they're one of the few models that has a clear path to resisting them.

[01:06:38]Linear models are just always going to be linear.

[01:06:40]They don't have much hope of resisting adversarial examples.

[01:06:44]Deep neural nets can be trained to be nonlinear, and so it seems like there's a path to a solution for them.

[01:06:50]Even with adversarial training, we still find that we aren't able to make models where if you optimize the input to belong to different classes, you get examples in those classes.

[01:07:01]Here I start with a CIFAR-10 truck and I turn it into each of the 10 different CIFAR-10 classes.

[01:07:07]Toward the middle of the plot you can see that the truck has started to look a little bit like a bird.

[01:07:12]But the bird class is the only one that we've come anywhere near hitting.

[01:07:15]So even with adversarial training, we're still very far from solving this problem.

[01:07:21]When we do adversarial training, we rely on having labels for all the examples.

[01:07:25]We have an image that's labeled as a bird.

[01:07:27]We make a perturbation that's designed to decrease the probability of the bird class, and we train the model that the image should still be a bird.

[01:07:33]But what if you don't have labels?

[01:07:35]It turns out that you can actually train without labels.

[01:07:39]You ask the model to predict the label of the first image.

[01:07:42]So if you've trained for a little while and your model isn't perfect yet, it might say, oh, maybe this is a bird, maybe it's a plane.

[01:07:47]There's some blue sky there, I'm not sure which of these two classes it is.

[01:07:51]Then we make an adversarial perturbation that's intended to change the guess and we just try to make it say, oh this is a truck, or something like that.

[01:07:59]It's not whatever you believed it was before.

[01:08:01]You can then train it to say that the distribution of our classes should still be the same as it was before, but this should still be considered probably a bird or a plane.

[01:08:10]This technique is called virtual adversarial training, and it was invented by Takeru Miyato.

[01:08:15]He was my Intern at Google after he did this work.

[01:08:18]At Google we invited him to come and apply his invention to text classification, because this ability to learn from unlabeled examples makes it possible to do semi-supervised learning where you learn from both unlabeled and labeled examples.

[01:08:35]And there's quite a lot of unlabeled text in the world.

[01:08:38]So we were able to bring down the error rate on several different text classification tasks by using this virtual adversarial training.

[01:08:47]Finally, there's a lot of problems where we'd like to use neural nets to guide optimization procedures.

[01:08:54]If we want to make a very, very fast car, we could imagine a neural net that looks at the blueprints for a car and predicts how fast it will go.

[01:09:02]If we could then optimize with respect to the input of the neural net and find the blueprint that it predicts would go the fastest, we could build an incredibly fast car.

[01:09:11]Unfortunately, what we get right now is not a blueprint for a fast car.

[01:09:14]We get an adversarial example that the model thinks is going to be very fast.

[01:09:18]If we're able to solve the adversarial example problem, we'll be able to solve this model-based optimization problem.

[01:09:25]I like to call model-based optimization the universal engineering machine.

[01:09:29]If we're able to do model-based optimization, we'll be able to write down a function that describes a thing that doesn't exist yet but we wish that we had.

[01:09:37]And then gradient descent and neural nets will figure out how to build it for us.

[01:09:41]We can use that to design new genes and new molecules for medicinal drugs, and new circuits to make GPUs run faster and things like that.

[01:09:51]So I think overall, solving this problem could unlock a lot of potential technological advances.

[01:09:58]In conclusion, attacking machine learning models is extremely easy, and defending them is extremely difficult.

[01:10:03]If you use adversarial training you can get a little bit of a defense, but there's still many caveats associated with that defense.

[01:10:11]Adversarial training and virtual adversarial training also make it possible to regularize your model and even learn from unlabeled data so you can do better on regular test examples, even if you're not concerned about facing an adversary.

[01:10:23]And finally, if we're able to solve all of these problems, we'll be able to build a black box model-based optimization system that can solve all kinds of engineering problems that are holding us back in many different fields.

[01:10:35]I think I have a few minutes left for questions.

[01:10:39][audience applauds] [speaker drowned out] Yeah.

[01:11:15]Oh, so, there's some determinism to the choice of those 50 directions.

[01:11:22]Oh right, yeah.

[01:11:23]So repeating the questions.

[01:11:24]I've said that the same perturbation can fool many different models or the same perturbation can be applied to many different clean examples.

[01:11:31]I've also said that the subspace of adversarial perturbations is only about 50 dimensional, even if the input dimension is 3,000 dimensional.

[01:11:40]So how is it that these subspaces intersect?

[01:11:43]The reason is that the choice of the subspace directions is not completely random.

[01:11:49]It's generally going to be something like pointing from one class centroid to another class centroid.

[01:11:55]And if you look at that vector and visualize it as an image, it might not be meaningful to a human just because humans aren't very good at imagining what class centroids look like.

[01:12:06]And we're really bad at imagining differences between centroids.

[01:12:10]But there is more or less this systematic effect that causes different models to learn similar linear functions, just because they're trying to solve the same task.

[01:12:22][speaker drowned out] Yeah, so the question is, is it possible to identify which layer contributes the most to this issue?

[01:12:33]One thing is that if you, the last layer is somewhat important.

[01:12:39]Because, say that you made a feature extractor that's completely robust to adversarial perturbations and can shrink them to be very, very small, and then the last layer is still linear.

[01:12:51]Then it has all the problems that are typically associated with linear models.

[01:12:57]And generally you can do adversarial training where you perturb all the different layers, all the hidden layers as well as the input.

[01:13:04]In this lecture I only described perturbing the input because it seems like that's where most of the benefit comes from.

[01:13:09]The one thing that you can't do with adversarial training is perturb the very last layer before the softmax, because that linear layer at the end has no way of learning to resist the perturbations.

[01:13:18]Doing adversarial training at that layer usually just breaks the whole process.

[01:13:23]But other than that, it seems very problem dependent.

[01:13:27]There's a paper by Sara Sabour and her collaborators called Adversarial Manipulation of Deep Representations, where they design adversarial examples that are intended to fool different layers of the net.

[01:13:41]They report some things about, like, how large of a perturbation is needed at the input to get different sizes of perturbation at different hidden layers.

[01:13:49]I suspect that if you trained the model to resist perturbations at one layer, then another layer would become more vulnerable and it would be like a moving target.

[01:14:00][speaker drowned out] Yes, so the question is, how many adversarial examples are needed to improve the misclassification rate?

[01:14:15]Some of our plots we include learning curves.

[01:14:20]Or some of our papers we include learning curves, so you can actually see, like in this one here.

[01:14:26]Every time we do an epoch we've generated the same number of adversarial examples as there are training examples.

[01:14:33]So every epoch here is 50,000 adversarial examples.

[01:14:37]You can see that adversarial training is a very data hungry process.

[01:14:43]You need to make new adversarial examples every time you update the weights.

[01:14:47]And they're constantly changing in reaction to whatever the model has learned most recently.

[01:14:55][speaker drowned out] Oh, the model-based optimization, yeah.

[01:15:11]Yeah, so the question is just to elaborate further on this problem.

[01:15:16]So most of the time that we have a machine learning model, it's something like a classifier or a regression model where we give it an input from the test set and it gives us an output.

[01:15:29]And usually that input is randomly occurring and comes from the same distribution as the training set.

[01:15:34]We usually just run the model, get its prediction, and then we're done with it.

[01:15:39]Sometimes we have feedback loops, like for recommender systems.

[01:15:44]If you work at Netflix and you recommend a movie to a viewer, then they're more likely to watch that movie and then rate it, and then there's going to be more ratings of it in your training set so you'll recommend it to more people in the future.

[01:15:57]So there's this feedback loop from the output of your model to the input.

[01:16:00]Most of the time when we build machine vision systems, there's no feedback loop from their output to their input.

[01:16:08]If we imagine a setting where we start using an optimization algorithm to find inputs that maximize some property of the output, like if we have a model that looks at the blueprints of a car and outputs the expected speed of the car, then we could use gradient ascent to look for the blueprints that correspond to the fastest possible car.

[01:16:31]Or for example if we're designing a medicine, we could look for the molecular structure that we think is most likely to cure some form of cancer, or the least likely to cause some kind of liver toxicity effect.

[01:16:45]The problem is that once we start using optimization to look for these inputs that maximize the output of the model, the input is no longer an independent sample from the same distribution as we used at the training set time.

[01:17:00]The model is now guiding the process that generates the data.

[01:17:06]So we end up finding essentially adversarial examples.

[01:17:11]Instead of the model telling us how we can improve the input, what we usually find in practice is that we've got an input that fools the model into thinking that the input corresponds to something great.

[01:17:23]So we'd find molecules that are very toxic but the model thinks they're very non-toxic.

[01:17:28]Or we'd find cars that are very slow but the model thinks are very fast.

[01:17:35][speaker drowned out] Yeah, so the question is, here the frog class is boosted by going in either the positive or negative adversarial direction.

[01:18:01]And in some of the other slides, like these maps, you don't get that effect where subtracting epsilon off eventually boosts the adversarial class.

[01:18:12]Part of what's going on is I think I'm using larger epsilon here.

[01:18:16]And so you might eventually see that effect if I'd made these maps wider.

[01:18:20]I made the maps narrower because it's like quadratic time to build a 2D map and it's linear time to build a 1D cross section.

[01:18:29]So I just didn't afford the GPU time to make the maps quite as wide.

[01:18:35]I also think that this might just be a weird effect that happened randomly on this one example.

[01:18:39]It's not something that I remember being used to seeing a lot of the time.

[01:18:43]Most things that I observe don't happen perfectly consistently.

[01:18:47]But if they happen, like, 80% of the time then I'll put them in my slide.

[01:18:52]A lot of what we're doing is trying trying to figure out more or less what's going on, and so if we find that something happens 80% of the time, then I consider it to be the dominant phenomenon that we're trying to explain.

[01:19:03]And after we've got a better explanation for that then I might start to try to explain some of the weirder things that happen, like the frog happening with negative epsilon.

[01:19:15][speaker drowned out] I didn't fully understand the question.

[01:19:24]It's about the dimensionality of the adversarial?

[01:19:34]Oh, okay.

[01:19:35]So the question is, how is the dimension of the adversarial subspace related to the dimension of the input?

[01:19:40]And my answer is somewhat embarrassing, which is that we've only run this method on two datasets, so we actually don't have a good idea yet.

[01:19:49]But I think it's something interesting to study.

[01:19:53]If I remember correctly, my coauthors open sourced our code.

[01:19:57]So you could probably run it on ImageNet without too much trouble.

[01:20:02]My contribution to that paper was in the week that I was unemployed between working at OpenAI and working at Google, so I had access to no GPUS and I ran that experiment on my laptop on CPU, so it's only really small datasets. [chuckles] [speaker drowned out] Oh, so the question is, do we end up perturbing clean examples to low confidence adversarial examples?

[01:20:47]Yeah, in practice we usually find that we can get very high confidence on the output examples.

[01:20:53]One thing in high dimensions that's a little bit unintuitive is that just getting the sign right on very many of the input pixels is enough to get a really strong response.

[01:21:06]So the angle between the weight vector matters a lot more than the exact coordinates in high dimensional systems.

[01:21:18]Does that make enough sense?

[01:21:20]Yeah, okay.

[01:21:21]- [Man] So we're actually going to [mumbles].

[01:21:23]So if you guys need to leave, that's fine.

[01:21:26]But let's thank our speaker one more time for getting-- [audience applauds]

Up Next

Intriguing Properties of Neural Networks Explained | CAP6412 Lecture Review

@UCFCRCV

1.1K views•2021-06-23

Building Real-Time ML Pipelines with Feature Stores and MLOps Frameworks

@ODSCAI

5.1K views•2022-02-20

Deep Reinforcement Learning Lecture: MDPs, Q-Learning, and Policy Gradients

@stanfordengineering

384.7K views•2017-08-11

Neural Networks Explained: Math, Layers, and Learning Fundamentals

@3blue1brown

21.9M views•2017-10-05

Related Study Plans & Knowledge Roadmaps

Structured learning paths in Artificial Intelligence

View All Artificial Intelligence Topics

artificial-intelligence

Building Recommenders: Two-Tower & Vector DBs

01Foundations of Python & Machine Learning Basics

02Introduction to Recommender Systems

03Deep Learning & Representation Learning

+ 3 more learning modules

Explore Roadmap

artificial-intelligence

Latent Diffusion: PyTorch, UNet & CLIP

01Deep Learning & PyTorch Foundations

02Convolutional Networks & Autoencoders

03The Math and Mechanics of Diffusion Models

+ 3 more learning modules

Explore Roadmap

artificial-intelligence

NeRFs: 3D Reconstruction & Instant-NGP

01Math & Python Foundations for 3D Vision

02Photogrammetry & Camera Pose Estimation

03Neural Radiance Fields (NeRF) Core Concepts

+ 2 more learning modules

Explore Roadmap

artificial-intelligence

TFT Forecasting: Energy Demand & Deep Learning

01Programming & Data Science Foundations

02Introduction to Machine Learning & Time Series

03Deep Learning & Sequential Models

+ 3 more learning modules

Explore Roadmap