Formal Verification in the Age of AI: Ensuring Safety, Security, and Correctness

Latest 50 papers on formal verification: Dec. 7, 2025

The relentless march of AI, particularly with the advent of Large Language Models (LLMs) and complex autonomous systems, has ushered in an era of unprecedented capabilities. However, this power comes with a critical challenge: ensuring these systems are safe, secure, and perform exactly as intended. This is where formal verification steps in, providing mathematical rigor to guarantee correctness. Recent research showcases exciting breakthroughs in bridging the gap between cutting-edge AI and robust formal methods. Let’s dive into some of the most compelling advancements.

The Big Idea(s) & Core Innovations: Bringing Rigor to AI

At the heart of these innovations is a common drive: to embed provable guarantees into increasingly complex and often opaque AI systems. Many papers tackle the inherent unpredictability of LLMs by integrating them with symbolic reasoning. For instance, The 4/δ Bound: Designing Predictable LLM-Verifier Systems for Formal Method Guarantee by Pierre Dantas, Lucas Cordeiro, Youcheng Sun, and Waldir Junior from the University of Manchester, UK, offers a theoretical framework based on Markov Chains to guarantee the convergence and termination of LLM-assisted verification. This work provides a crucial δ parameter, allowing engineers to quantify verification success and plan resources systematically.

Expanding on the integration of LLMs, SHIELDAGENT: Shielding Agents via Verifiable Safety Policy Reasoning from Zhaorun Chen, Mintong Kang, and Bo Li at the University of Chicago and the University of Illinois at Urbana-Champaign, introduces a novel guardrail agent. This agent enforces safety policy compliance for autonomous agents through probabilistic logic reasoning, effectively safeguarding LLM-based agents from malicious instructions and adversarial attacks. Similarly, Beyond Prompt Engineering: Neuro-Symbolic-Causal Architecture for Robust Multi-Objective AI Agents by Gokturk Aytug Akarlar proposes the Chimera architecture, which marries neural reasoning with formal verification (using TLA+) and causal inference, demonstrating significant improvements in reliability for LLM agents compared to prompt engineering alone.

Formal verification is also making significant strides in critical domains like hardware design and control systems. VeriThoughts: Enabling Automated Verilog Code Generation using Reasoning and Formal Verification, from NYU Tandon School of Engineering, introduces a dataset and benchmark framework to evaluate LLM-generated hardware descriptions using formal verification instead of traditional simulations. This is complemented by ProofWright: Towards Agentic Formal Verification of CUDA by Bodhisatwa Chatterjee et al. from Georgia Institute of Technology and NVIDIA Research, which formally verifies LLM-generated CUDA code for correctness and safety, ensuring thread and memory safety for GPU kernels. On the control systems front, Robust Verification of Controllers under State Uncertainty via Hamilton-Jacobi Reachability Analysis by Albert Lin et al. from Stanford University and NASA Jet Propulsion Laboratory presents RoVer-CoRe, the first Hamilton-Jacobi (HJ) reachability-based framework for verifying perception-based systems under perceptual uncertainty.

Safety in robotics and autonomous systems is a recurring theme. The paper Formal Verification of Probabilistic Multi-Agent Systems for Ballistic Rocket Flight Using Probabilistic Alternating-Time Temporal Logic by Damian Kurpiewski et al. from the Polish Academy of Sciences details a framework for analyzing safety in ballistic rocket flight, using PATL to account for environmental stochasticity. In a similar vein, Formal Verification of Noisy Quantum Reinforcement Learning Policies by Dennis Gross (LAVA Lab) introduces QVerifier to verify quantum reinforcement learning (QRL) policies against safety properties, even accounting for quantum noise. The flexibility of formal methods is further demonstrated by VeriODD: From YAML to SMT-LIB – Automating Verification of Operational Design Domains by Bassel Rafie from RWTH Aachen University, which automates the verification of operational design domains (ODDs) for autonomous driving by translating human-readable specifications into formal constraints.

Under the Hood: Models, Datasets, & Benchmarks

These advancements are often powered by novel tools, datasets, and benchmarks that enable rigorous evaluation and facilitate further research:

• VeriThoughts Dataset: Introduced in VeriThoughts: Enabling Automated Verilog Code Generation using Reasoning and Formal Verification, this is the first large-scale dataset of over 20,000 Verilog modules with paired prompts/questions and reasoning traces, specifically designed for LLM-based hardware code generation. Its validation uses formal verification, marking a shift from traditional testbench simulations.
• SHIELDAGENT-BENCH: From SHIELDAGENT: Shielding Agents via Verifiable Safety Policy Reasoning, this is a comprehensive benchmark dataset for evaluating agent guardrails across diverse web environments and risk categories.
• BarrierBench: Featured in BarrierBench : Evaluating Large Language Models for Safety Verification in Dynamical Systems, this benchmark comprises 100 dynamical systems across various domains and complexities, used to evaluate LLMs for synthesizing safety certificates.
• ConstructiveBench Dataset: Enumerate-Conjecture-Prove: Formally Solving Answer-Construction Problems in Math Competitions introduces this autoformalized dataset of 3,640 math competition problems with verified Lean formalizations, available on GitHub and Hugging Face for benchmarking formal mathematical reasoning (https://github.com/sunjia72/ECP, https://huggingface.co/datasets/sunjia72/ConstructiveBench).
• CoqDev Benchmark: Proposed in Adaptive Proof Refinement with LLM-Guided Strategy Selection, this new benchmark is mined from real-world Coq commit histories to model incremental development processes in theorem proving. The code for Adapt, their LLM-based framework, is available at https://github.com/purdue-adapt/Adapt.
• VeriStruct Tool: Described in VeriStruct: AI-assisted Automated Verification of Data-Structure Modules in Verus, this tool implements a workflow for generating program verification annotations for complex data-structure modules in Rust using Verus. The code is available at https://anonymous.4open.science/r/FVagent-Artifact-0653 and https://github.com/verus-lang/verus.
• VeriODD Tool: From VeriODD: From YAML to SMT-LIB – Automating Verification of Operational Design Domains, this tool converts YAML-based ODD descriptions into SMT-LIB format for automated formal verification of autonomous driving systems. Its code can be found at https://github.com/BasselRafie/VeriODD.
• Galapagos Framework: Galapagos: Automated N-Version Programming with LLMs introduces this framework for automated N-Version programming with formal guarantees using LLMs, available open-source at https://github.com/ASSERT-KTH/Galapagos/.
• RocqStar: The framework for generating proofs in the Rocq interactive theorem prover using similarity-driven retrieval and agentic systems, with code available at https://github.com/JetBrains-Research/rocqstar-agentic-system/tree/main/mcpServer and https://github.com/JetBrains-Research/rocqstar-agentic-system/tree/main/koogAgent.

Impact & The Road Ahead

These advancements have profound implications for the future of AI/ML. We are moving towards a paradigm where AI systems, particularly LLM-powered agents, are not just intelligent but also provably reliable. This research enables a new generation of predictable LLM-verifier systems, secure LLM-generated code, and robust autonomous agents in safety-critical domains like aerospace, robotics, and smart contracts.

The integration of LLMs with formal methods, as seen in LangSAT: A Novel Framework Combining NLP and Reinforcement Learning for SAT Solving and Automated Generation of MDPs Using Logic Programming and LLMs for Robotic Applications, promises to make complex logic-based automation more accessible and interpretable. Furthermore, the ability to formally verify quantum reinforcement learning policies, as showcased by QVerifier, is critical for the nascent but rapidly growing field of quantum computing.

The concept of continuous assurance, as highlighted in Towards Continuous Assurance with Formal Verification and Assurance Cases, is crucial for maintaining trustworthiness throughout the lifecycle of autonomous systems. Papers like Towards a Formal Verification of Secure Vehicle Software Updates and Quantum-Resistant Authentication Scheme for RFID Systems Using Lattice-Based Cryptography underscore the increasing importance of formal guarantees in cybersecurity and IoT, especially against emerging quantum threats.

While challenges remain, particularly in scaling formal methods to ever-larger and more complex AI systems, the progress is undeniable. The future lies in intelligent agentic systems that can not only generate powerful solutions but also prove their correctness and safety. This convergence of AI’s generative power with formal methods’ rigorous guarantees is paving the way for truly trustworthy and transformative technologies.