Physics-Informed Neural Networks: Architectures, Optimizers, and Scalability Breakthroughs
Latest 13 papers on physics-informed neural networks: Oct. 3, 2026
Physics-InInformed Neural Networks (PINNs) have revolutionized scientific machine learning by embedding governing physical laws directly into neural network training. This unique capability allows them to solve complex differential equations and discover hidden physics from data, even when data is sparse. However, challenges like scalability, handling stiff and high-dimensional problems, ensuring physical consistency, and optimizing their training have been persistent hurdles. Recent research, synthesized from a collection of innovative papers, reveals exciting breakthroughs that are propelling PINNs into new frontiers.
The Big Idea(s) & Core Innovations
At the heart of these advancements is a multi-pronged attack on PINN limitations. One major theme is the quest for more efficient and stable optimization. The paper, “SoftServe: A Scalable Quasi-Newton Method for Deep Learning” by Joohwan Ko et al. (University of Massachusetts Amherst, CCM, Flatiron Institute, Cornell University), introduces SOFTSERVE. This quasi-Newton optimization framework tackles non-convexity and scalability by deriving positive-definite curvature estimates without line searches, crucially applying Kronecker-factored and diagonal structures to scale to massive networks. This is particularly valuable for ill-conditioned tasks, including PINNs themselves. Complementing this, Thomas Borsani and Giuseppe Di Fatta (Free University of Bozen – Bolzano, Italy) in “Gradient Surgery for Physics-Informed Neural Networks” systematically study gradient conflicts in PINNs, proposing PAM-GS (Physics-Aware Momentum Gradient Surgery). They ingeniously adapt conflict mitigation strategies based on distinct training phases, leading to significant performance improvements by explicitly handling both angle-based and magnitude-based gradient conflicts.
Another dominant thread is enhancing PINN architectures for specific challenges. For high-order partial differential equations (PDEs), B. Veena S. N. Rao (Texas A&M University-Corpus Christi, USA) presents “Mesh-Free Numerical Approximation of the Biharmonic Equation via Optimized Kolmogorov–Arnold Neural Networks”. This work leverages Kolmogorov-Arnold Networks (KANs) with learnable univariate functions on network edges, achieving remarkable accuracy for the biharmonic equation with a minimal parameter count. Meanwhile, for handling stiff PDEs and inverse problems, Márcio Marques et al. (Instituto de Matemática Pura e Aplicada (IMPA), Brazil) introduce NEXT: “Physics-Informed Neuro-Spectral Exponential Time Differencing Architectures”. NEXT cleverly integrates the linear stiff part of PDEs analytically via matrix exponentials within a Neuro-Spectral Architecture, offering superior stability and accuracy. In a similar vein of architectural innovation, Xiaodong Feng et al. (Beijing Normal-Hong Kong Baptist University, China; Beijing Normal University, China; Guangzhou Nanfang College, China; Louisiana State University, USA; Chinese Academy of Sciences) propose “CI-PINN: Causal Integral Physics-Informed Neural Network for Solving Evolution Equations”, embedding temporal causality directly into the network architecture through Volterra-type causal integral terms, outperforming standard PINNs in accuracy and robustness, especially in sparse-collocation scenarios.
Scalability and high-dimensional problems receive significant attention. Huiwen Zhang et al. (University of Wisconsin-Madison, USA) tackle the scalability bottleneck with “PE-EK-PINN: Physics Embedding with Evolving Kernel for Scalable Physics-Informed Neural Networks”. This hierarchical framework reuses learned subsystem fields as “evolved kernels” for larger configurations, dramatically reducing cumulative training costs from O(N) to O(log N). For parametric PDEs in high dimensions, Denis Korolev and Martin Eigel (Weierstrass Institute for Applied Analysis and Stochastics (WIAS), Germany) introduce “Tensor-Train Compressed Separable PINNs: A Curvature-Aware Optimization Framework for Parametric PDEs in High Dimensions”. This framework uses tensor-train compression and separable neural architectures to achieve exact compressed Gauss-Newton steps, circumventing the curse of dimensionality by ensuring effective compressed dimension grows linearly, not exponentially.
Finally, ensuring physical consistency beyond just fitting equations is explored. Tuan Luong and Hyungpil Moon (Sungkyunkwan University, South Korea) propose DissipNet in “Data-driven discrete-time deep recurrent neural network-based modeling for dissipative systems”. Unlike PINNs that use physics as soft regularization, DissipNet explicitly enforces dissipativity through structural weight constraints and Lyapunov theory, providing explicit stability guarantees critical for physical AI. For improving the representation of oscillatory phenomena, Michael Mommert et al. (German Aerospace Center (DLR), Germany; Technische Universität Ilmenau, Germany) in “An improved periodic activation for PINNs reconstructing convective flows” develop a novel PINN architecture using complex exponential activation functions to generate paired sine-cosine outputs, significantly improving temperature field reconstruction in convective flows. In the realm of inverse problems with uncertainty, Iraklis Spyrou et al. (INSANE Group, IIT, NCSR Demokritos, Greece; University of Thessaly, Greece) introduce QUPI-PINN in “Probabilistic Physics-Informed Neural Solvers for Woods-Saxon Parameter Identification: A Coupled Forward-Inverse Approach” for nuclear mean-field potentials. This framework couples forward eigenvalue solving with inverse parameter identification, recovering six global Woods-Saxon parameters with sub-percent accuracy from sparse spectral data and providing uncertainty quantification.
Under the Hood: Models, Datasets, & Benchmarks
These papers highlight a diverse set of computational tools and evaluation strategies:
- Optimizers: SOFTSERVE introduces diagonal and Kronecker-factored variants with GPU-friendly Newton-Schulz iterations. PAM-GS is a novel gradient surgery method that adaptively detects and mitigates angle-based and magnitude-based gradient conflicts.
- Architectures: KAN-PINNs (Kolmogorov-Arnold Neural Networks) with learnable univariate functions are shown to be highly effective for high-order PDEs. CI-PINN leverages a Causal Integral Neural Network (CinNet) for temporal causality. NEXT combines Neuro-Spectral Architectures (NeuSA) with exponential time differencing (ETD-RK4) for stiff PDEs. QUPI-PINN integrates WaveNet for wavefunction representation and ParamNet for probabilistic inference.
- Compression Techniques: Tensor-Train (TT) and Canonical Polyadic (CP) formats are used for compressing representations in high-dimensional parametric PDEs.
- Datasets & Benchmarks: Experiments span a wide range of scientific and engineering problems: ill-conditioned tasks (RNNs, autoencoders), biharmonic equation (thin plate deflection), Helmholtz equation (wave propagation, antenna arrays), stochastic delayed differential equations (SDDEs) using CausalDynamics and Dysts benchmarks, various evolution equations (Allen-Cahn, KdV, Cahn-Hilliard), high-dimensional parametric PDEs, stiff PDEs (heat, wave, Burgers, KdV equations), infinity and p-Laplace problems, Rayleigh-Bénard convection, Woods-Saxon nuclear mean-field potentials (synthetic and experimental data), and medical treatment environments like sepsis and acute hypotension using MIMIC-III and synthetic datasets within MedGym.
- Code Repositories: Several authors have made their work publicly accessible:
- SOFTSERVE: https://github.com/joohwanko/SoftServe
- L0-NDDE (for SDDEs): https://anonymous.4open.science/r/ODEDriverIdentifiability-C325/README.md
- NEXT: https://github.com/marcioh2m/next.git
- PAM-GS (Gradient Surgery for PINNs): https://github.com/ThomBors/gs_pinn
- MedGym: https://github.com/wangadam782/MedGym
Impact & The Road Ahead
These breakthroughs have profound implications for scientific machine learning and real-world applications. The improved optimization techniques (SOFTSERVE, PAM-GS) promise more robust and faster training for PINNs across the board. Architectural innovations like KAN-PINNs and NEXT open doors to efficiently solving previously intractable high-order and stiff PDEs. The hierarchical kernel learning in PE-EK-PINN and tensor-train compression in “Tensor-Train Compressed Separable PINNs” fundamentally address the scalability challenges that have limited PINN adoption for large-scale, complex systems. This is critical for fields like electromagnetics, climate modeling, and uncertainty quantification.
The explicit enforcement of physical properties, as seen in DissipNet for dissipative systems and CI-PINN for temporal causality, ensures that learned models are not just accurate but also physically consistent and stable, vital for safety-critical applications like robotics and control systems. The ability to identify drivers of delayed physical systems with identifiability guarantees, as shown by Julien Boussard et al. (McGill University, Canada; Mila – Quebec AI Institute, Canada; Université Évry Paris-Saclay, France; Université de Montréal, Canada) in “Identifiability Guarantees for Drivers and Dynamics of Delayed Physical Systems”, has significant implications for modeling complex biological or environmental systems. Furthermore, the MedGym benchmark, developed by Yuepeng Wang et al. (Tokyo University of Agriculture and Technology, Japan; Institute of Science Tokyo, Japan; National University of Singapore, Singapore; LY Corporation, Japan; University of Toronto, Canada; National Institute of Advanced Industrial Science and Technology (AIST), Japan; Emory University, USA; Norwegian University of Science and Technology, Norway), highlights the critical role of PINNs in developing personalized and safe medical treatments via continuous-time reinforcement learning.
Looking ahead, these advancements suggest a future where PINNs are not only more accurate and efficient but also inherently more physically grounded and scalable. The convergence of advanced optimization, specialized architectures, and robust physical constraints is paving the way for PINNs to tackle grand challenges in scientific discovery and engineering, from designing next-generation materials to understanding complex biological processes. The journey towards fully autonomous scientific AI is accelerating, driven by these relentless innovations in physics-informed machine learning.
Share this content:
Discover more from SciPapermill
Subscribe to get the latest posts sent to your email.
Post Comment