Solving the Heat Equation with DeepXDE and PINNs

For inverse problems, PINNs treat unknown differential equation parameters as learnable variables alongside neural network weights. The boundary loss is replaced with a data loss matching observations at arbitrary domain points. During training, both the solution network and equation parameters are optimized simultaneously. This joint optimization approach differs fundamentally from traditional methods that alternate between forward simulation and parameter estimation. The method requires sufficient observational data to uniquely determine parameters and careful handling of dimensional consistency between loss terms.

Physics-Informed Neural Networks (PINNs) are a class of machine learning models that incorporate physical laws (expressed as partial differential equations) directly into their loss functions, enabling them to solve both forward problems (simulating known PDEs) and inverse problems (learning unknown parameters or equations from data) without requiring extensive computational grids; PINNs achieve this by training neural networks to minimize a composite loss function that includes terms enforcing PDE residuals and boundary/initial conditions, with automatic differentiation handling the complex derivative calculations needed for PDE constraints.

PINNs solve inverse problems by treating unknown physical parameters as learnable parameters alongside neural network weights. The loss function combines physics loss (enforcing differential equations) and data loss (matching noisy observations). This physics regularization prevents overfitting to sparse/noisy measurements, producing smooth physically-consistent solutions. Implementation in PyTorch involves defining learnable parameters as torch.nn.Parameter objects, configuring optimizers to update both network weights and physical parameters, and monitoring convergence. Practical applications include estimating material properties from experimental data and discovering governing equations from observations. Future directions involve hybrid approaches combining PINNs with traditional numerical methods, addressing scaling challenges in higher dimensions through domain decomposition, and developing better theoretical understanding of PINN behavior. Current research trends include weak formulation PINNs, equation discovery techniques, and tighter integration with finite element/difference methods.

Inverse Physics-Informed Neural Networks (PINNs) estimate unknown parameters of partial differential equations by treating these parameters as learnable variables within a neural network, where the training process minimizes a combined loss function that enforces both the PDE constraints and data fitting accuracy; this approach was demonstrated by estimating the diffusion coefficient (lambda_2) of the Burgers equation to approximately 0.2034 (compared to the true value of 0.2031) with a relative error of only 1.3% on the solution.

This segment demonstrates PINN capabilities for interpolation and inverse problems. The presenter explains that PINNs can predict solutions throughout an entire domain given only boundary and initial conditions, without requiring intermediate data. The presenter also explains how PINNs can solve inverse problems, where the goal is to determine unknown parameters (like thermal diffusivity) from observed data. This capability makes PINNs valuable for characterizing materials and systems where direct measurement of parameters is difficult.

Scientific machine learning combines numerical analysis and machine learning for scientific computing challenges. Three main categories exist: forward problems (PINNs) where PDEs are known but data is scarce, inverse problems (SINDy) where coefficients are unknown but some terms are known, and operator learning where neither equations nor physics knowledge is available. Operator learning approximates unknown differential operators L mapping sources to solutions. Multiple neural architectures address this: Branch and Trunk Networks perform low-rank approximations, Fourier Neural Operators use FFT and convolutions (spectral methods), DeepONets employ non-code architectures, and Transformers have been explored. Theoretical foundations include universal approximation theorems stating neural operators can represent operators between function spaces, and complexity bounds quantifying parameter requirements. Neural operators trained at one resolution can evaluate at different resolutions without significant error increase. The connection between neural operators and numerical linear algebra shows architectures correspond to specific matrix structures: FNOs enforce spectral matrices, DeepONets use low-rank approximations.

Two dominant neural operator paradigms emerged: DeepONets use linear combinations of basis functions with separate trunk and branch networks, while FNOs employ FFT-based integral operators with implicitly periodic kernels. Both approaches represent the same underlying problem but make different assumptions about kernel structure. A critical tension exists between accuracy and interpretability: neural operators offer high expressiveness but with millions of parameters, while interpolation-based methods provide transparency and error guarantees at potential cost of flexibility. This interpretability-accuracy slider represents a fundamental design choice in scientific machine learning, with implications for trustworthiness and generalizability of surrogate models.

This section analyzes the trade-offs between different neural operator architectures. Three critical properties distinguish them: discretization invariance (inputting functions at different positions without retraining), prediction freedom (evaluating output at arbitrary positions), and domain independence (mapping between different dimensional domains). DeepONet is prediction-free but not discretization-invariant. Fourier Neural Operators are discretization-invariant but not prediction-free. Benchmark studies show that with proper extensions, both architectures can achieve similar accuracy for comparable parameter counts. BelNet (2022) generalizes DeepONet to achieve both properties, with DeepONet and FNOs as special cases.

Functions map numbers to numbers (expressible by standard ANNs), while operators map functions to functions. DeepONet addresses operator learning by using a basis function and coefficient framework. The trunk network outputs basis functions, while the branch network outputs coefficients. The final output is a linear combination of basis functions weighted by coefficients. This perspective shows DeepONet learns a function representation using a learned basis set, allowing standard ANN structures to handle operator learning without new architectures.

Neural Operator Learning learns mappings between function spaces rather than single solution instances. DeepONet uses two sub-networks: a trunk network processing spatial coordinates (x, t) and a branch network processing input functions (boundary conditions, forcing functions). Outputs combine through inner product operations, enabling prediction for new input conditions without retraining. Fourier Neural Operator (FNO) uses Fourier transforms to learn mappings, with encoder, processing layer, and decoder components. FNO uses skip connections similar to U-Net architectures. Both approaches enable efficient solution prediction across varying input conditions, representing a paradigm shift from instance-based to function-based learning in computational physics.

Several advanced strategies improve PINN training stability and efficiency: finite difference approximations for gradient computation offer modest speedups over automatic differentiation; hard constraints encode boundary conditions directly into the solution ansatz, eliminating competition between loss terms; self-adaptive weighting treats loss coefficients as learnable parameters, concentrating training effort on high-residual regions; adaptive sampling dynamically concentrates collocation points where the PDE residual is largest, mimicking adaptive mesh refinement. These methods transform difficult optimization problems into more tractable forms, though they introduce additional complexity in implementation and hyperparameter management.

Self-Adaptive PINN (SA-PINN) addresses manual weight tuning by using mask functions that assign adaptive weights to each training point, updated via gradient ascent. Adaptive activation functions incorporate trainable parameters within activation functions (e.g., hyperbolic tangent with scaling parameters) to improve training speed. Combining adaptive loss weighting with adaptive activation functions provides significantly better convergence than either technique alone. These techniques address common PINN training challenges including difficulty balancing loss components and achieving fast convergence.

Physics-Informed Neural Networks (PINNs) are coordinate-based neural networks where input is physical coordinates and time, output is the solution field. Using automatic differentiation, PINNs compute PDE residuals and minimize losses across initial/boundary conditions and collocation points. However, PINNs suffer from a fundamental failure mode: loss component imbalance causes convergence to physically incorrect solutions that satisfy PDEs but violate initial conditions. Self-adaptive PINNs address this by introducing adaptive weights that automatically emphasize critical regions during training, enabling successful solutions for stiff equations like Allen-Cahn that standard PINNs fail to converge on. PINNs also enable parameter estimation by treating unknown coefficients as optimization variables alongside network weights, successfully identifying parameters from noisy data through self-adaptive weighting mechanisms.

Gradient descent optimization in PINNs faces stiffness-related challenges analogous to those in traditional ODE solvers. Gradient descent itself can be viewed as Euler's method applied to an ODE defined by the gradient in parameter space. When the loss landscape exhibits steep gradients (stiffness), standard gradient descent fails similarly to explicit ODE solvers failing on stiff equations. Research shows that standard neural network architectures naturally lack the eigenvalue separation needed to handle stiff problems effectively. Very few parameter combinations produce solutions with appropriate time-scale separation, making training difficult. This suggests that new neural network architectures specifically designed for stiff problems, combined with specialized optimizers, will be necessary for PINNs to achieve the same robustness that stiff solvers like ODE15s provide for traditional numerical methods.

PINN training faces fundamental challenges requiring advanced techniques. Dynamic loss weighting adjusts relative weights based on gradient magnitudes to balance PDE residual and boundary condition losses. Neural tangent matrix analysis shows PDE residual sensitivity eigenvalues are much larger than boundary condition sensitivities, explaining training difficulties. Curriculum regularization addresses complexity by gradually increasing PDE coefficients during training, starting with simpler problems. GPINN incorporates additional constraints on PDE residual gradients, adding another loss term for zero-slope enforcement. Region optimization prevents overfitting by sampling entire regions rather than fixed points, distributing optimization effort across domains. PINN offers advantages over traditional numerical methods (faster for certain problems, handles complex geometries) and pure deep learning (generalization with limited data, incorporates physical knowledge, enables unsupervised inverse learning). Ongoing research focuses on analyzing and resolving these limitations to improve reliability and applicability.