OCR’s Next Chapter: Vision-Language Models, Robustness, and Real-World Applications

Latest 46 papers on optical character recognition: Nov. 2, 2025

Optical Character Recognition (OCR) has long been a cornerstone of digital transformation, converting static images into editable, searchable text. Yet, as AI models become more sophisticated, the challenges facing OCR have also evolved—from deciphering historical scripts and complex engineering drawings to navigating adversarial attacks and enhancing multimodal understanding. Recent research highlights a significant shift: moving beyond mere text extraction to a deeper, context-aware interpretation, often powered by the synergy of vision and language models (VLMs).

The Big Idea(s) & Core Innovations

At the heart of recent breakthroughs is the integration of visual context and advanced language understanding. One major theme is the quest for more accurate and robust text recognition, particularly in challenging scenarios. For instance, the paper “GLYPH-SR: Can We Achieve Both High-Quality Image Super-Resolution and High-Fidelity Text Recovery via VLM-guided Latent Diffusion Model?” by Mingyu Sung et al. addresses the critical problem of restoring legible scene-text in low-quality images, where traditional super-resolution often fails. Their GLYPH-SR model, combining perceptual quality and OCR accuracy, significantly improves OCR F1 scores by over 15 percentage points by employing a dual-branch Text-SR Fusion ControlNet. This demonstrates a move towards objective-driven image restoration that prioritizes text legibility.

Another significant innovation focuses on OCR-free or OCR-enhanced multimodal systems. For specialized domains, traditional OCR can be a bottleneck. Researchers from A*STAR and Nanyang Technological University, Singapore, in their paper “A Multi-Stage Hybrid Framework for Automated Interpretation of Multi-View Engineering Drawings Using Vision Language Model”, introduce a three-stage, OCR-free framework leveraging fine-tuned Donut-based VLMs. This framework significantly improves accuracy and scalability in extracting structured information from complex engineering drawings, bypassing the limitations of generic OCR. Similarly, “See then Tell: Enhancing Key Information Extraction with Vision Grounding” by Shuhang Liu et al. introduces STNet, an end-to-end model that uses a novel <see> token to implicitly encode spatial coordinates, integrating vision grounding directly into text generation for key information extraction, outperforming pure OCR or OCR-free methods.

The challenge of low-resource languages and historical documents also sees significant advancements. The “VOLTAGE: A Versatile Contrastive Learning based OCR Methodology for ultra low-resource scripts through Auto Glyph Feature Extraction” by Prawaal Sharma et al. (Infosys, BITS Pilani) introduces an unsupervised OCR methodology for scripts like Takri, achieving high accuracy with minimal manual intervention. “CHURRO: Making History Readable with an Open-Weight Large Vision-Language Model for High-Accuracy, Low-Cost Historical Text Recognition” from Stanford University presents CHURRO, a 3B-parameter VLM specialized for historical text, demonstrating superior performance and cost-effectiveness on diverse historical documents. Building on this, the paper “Improving OCR for Historical Texts of Multiple Languages” by Hylke Westerdijk et al. from the University of Groningen, explores deep learning methods to enhance OCR for Hebrew and English handwriting, showcasing the power of transformer-based models and data augmentation.

Furthermore, researchers are addressing the vulnerabilities and cascading effects of OCR errors. “When Vision Fails: Text Attacks Against ViT and OCR” by Nicholas Boucher et al. (University of Cambridge, Oxford, Toronto) highlights how subtle Unicode-based adversarial examples can fool OCR and Vision Transformers without affecting human readability, emphasizing the need for more robust defenses. Concurrently, “OCR Hinders RAG: Evaluating the Cascading Impact of OCR on Retrieval-Augmented Generation” introduces OHRBench, revealing that OCR noise significantly degrades Retrieval-Augmented Generation (RAG) systems, underscoring the inadequacy of current OCR solutions for high-quality knowledge bases.

Under the Hood: Models, Datasets, & Benchmarks

The innovations discussed are often underpinned by novel models, specialized datasets, and rigorous benchmarks:

• GLYPH-SR: A vision-language guided diffusion model using a Bi-objective formulation and a Text-SR Fusion ControlNet for balanced perceptual quality and OCR accuracy. The code is likely to be released at https://github.com/GLYPH-SR/GLYPH-SR.
• DocTron-Formula: A unified framework leveraging general vision-language models for formula recognition, supported by CSFormula, a challenging multidisciplinary dataset at various structural levels. Code is available at https://github.com/DocTron-hub/DocTron-Formula.
• CHURRO: A 3B-parameter open-weight VLM for historical text, alongside CHURRO-DS, the largest and most diverse dataset for historical OCR (over 99,491 pages across 46 language clusters). Code for CHURRO is provided by https://gith.
• Logics-Parsing: An end-to-end LVLM-based framework enhanced with reinforcement learning for layout-aware document parsing, with the comprehensive LogicsParsingBench benchmark. Code is available at https://github.com/alibaba/Logics-Parsing.
• DianJin-OCR-R1: A reasoning-and-tool interleaved VLM that combines LVLMs with expert OCR tools, demonstrating superior performance on benchmarks like ReST and OmniDocBench. Code is at https://github.com/aliyun/qwen-dianjin.
• OHRBench: The first benchmark specifically designed to evaluate the cascading impact of OCR errors on Retrieval-Augmented Generation (RAG) systems. The dataset and evaluation code are available at https://github.com/opendatalab/OHR-Bench.
• MultiOCR-QA: A new multilingual QA dataset derived from historical texts with OCR errors, used to evaluate LLM robustness against OCR noise. Code for dataset generation and evaluation will be released post-publication.
• iWatchRoad: A pothole detection system using a custom YOLO model fine-tuned on BharatPotHole, a large, self-annotated dataset for Indian road conditions. Code for iWatchRoad can be found at https://github.com/smlab-niser/iwatchroad.
• Uni-MuMER: A unified multi-task fine-tuning framework for handwritten mathematical expression recognition using VLMs, achieving state-of-the-art results on CROHME and HME100K datasets. Code is available at https://github.com/BFlameSwift/Uni-MuMER.
• DocIQ: A new benchmark dataset and feature fusion network for document image quality assessment. More details on the dataset can be found at https://arxiv.org/abs/2410.12628.
• SynthID: An end-to-end pipeline for generating high-fidelity synthetic invoice documents with paired image-JSON data, providing a solution to data scarcity. Code is available at https://github.com/BevinV/Synthetic_Invoice_Generation.
• From Press to Pixels: An OCR pipeline for Urdu newspapers, introducing the Urdu Newspaper Benchmark (UNB) dataset and leveraging SwinIR for super-resolution. Code includes fine-tuned YOLOv11x models and LLM-based text recognition.
• IVGocr and IVGdirect: Two visual grounding methods for GUI interaction, with a publicly released test dataset at https://arxiv.org/pdf/2407.01558.
• E-ARMOR: A framework for assessing multilingual OCR systems in edge cases, utilizing existing tools like Surya, PyLaia, and Kraken, with code at https://github.com/datalab-to/surya.
• Line-Level OCR: A new dataset of 251 English page images with line-level annotations, promoting a shift from word to line-level OCR. Website: https://nishitanand.github.io/line-level-ocr-website. The code for the system is likely based on https://github.com/mindee/doctr.
• Evaluating OCR performance on food packaging labels in South Africa: A comparative study benchmarking Tesseract, EasyOCR (https://github.com/JaidedAI/EasyOCR), PaddleOCR, and TrOCR on real-world food packaging images.

Impact & The Road Ahead

The implications of these advancements are profound. We are moving towards OCR systems that are not just transcription tools but intelligent document understanding agents. This transition enables robust automation in diverse fields, from industrial digital manufacturing with automated interpretation of engineering drawings to scalable geospatial search on historical maps using GPT-4o, as shown by Sofia Kirsanova et al. from the University of Minnesota in “Detecting Legend Items on Historical Maps Using GPT-4o with In-Context Learning”.

In digital humanities, the ability to accurately digitize historical texts, regardless of script or degradation, is a game-changer. The medical field benefits from precise DICOM de-identification, as seen in the hybrid AI-based approach by Hamideh Haghiri et al. (German Cancer Research Center) in “A Hybrid AI-based and Rule-based Approach to DICOM De-identification: A Solution for the MIDI-B Challenge”, and the potential for digital assessments with explainable AI grading by TrueGradeAI, by Rakesh Thakur et al. (Amity University) in “TrueGradeAI: Retrieval-Augmented and Bias-Resistant AI for Transparent and Explainable Digital Assessments”. Even sports analytics are being transformed, with OCR-guided YOLOv8 detecting wicket-taking deliveries in cricket videos, as explored in “Automated Wicket-Taking Delivery Segmentation and Weakness Detection in Cricket Videos Using OCR-Guided YOLOv8 and Trajectory Modeling”.

The road ahead involves creating even more robust, context-aware, and ethically sound OCR and document intelligence systems. Addressing vulnerabilities to adversarial attacks, improving performance in ultra-low-resource languages, and mitigating the cascading impact of OCR errors on downstream AI applications like RAG will be crucial. The rise of hybrid approaches, combining the reasoning power of VLMs with specialized expert tools, suggests a promising path towards AI that not only sees text but truly understands its context and implications. The continuous innovation in models, datasets, and evaluation frameworks promises a future where digitizing and interpreting information from any visual source is seamless and reliable.