Publications

REAR: Retrieve, Expand and Refine for Effective Multitable Retrieval

Rishita Agarwal*, Himanshu Singhal*, Peter Baile Chen, Manan Roy Choudhury, Dan Roth, Vivek Gupta ACL 2026

Answering natural language queries over re- lational data often requires retrieving and rea- soning over multiple tables, yet most retrievers optimize only for query–table relevance and ignore table–table compatibility. We introduce REAR (Retrieve, Expand and Refine), a three- stage, LLM-free framework that separates se- mantic relevance from structural joinability for efficient, high-fidelity multi-table retrieval. REAR (i) retrieves query-aligned tables, (ii) ex- pands these with structurally joinable tables via fast, precomputed column-embedding compar- isons, and (iii) refines them by pruning noisy or weakly related candidates. Empirically, REAR is retriever-agnostic and consistently improves dense/ sparse retrievers on complex table QA datasets (BIRD, MMQA, and Spider) by im- proving both multi-table retrieval quality and downstream SQL execution. Despite...

paper code

CRAFT: Training-Free Cascaded Retrieval for Tabular QA

Adarsh Singh*, Kushal Raj*, Bhandari, Jianxi Gao, Soham Dan, Vivek Gupta ACL 2026

Open-Domain Table Question Answering (TQA) involves retrieving relevant tables from a large corpus to answer natural language queries. Traditional dense retrieval mod- els, such as DTR and DPR, not only incur high computational costs for large-scale re- trieval tasks but also require retraining or fine- tuning on new datasets, limiting their adapt- ability to evolving domains and knowledge. In this work, we propose CRAFT, a zero- shot, cascaded retrieval approach that first uses a sparse retrieval model to filter a subset of candidate tables before applying more com- putationally expensive dense models as re- rankers. To improve retrieval quality, we en- rich table representations with descriptive ti- tles and summaries generated by Gemini Flash 1.5, enabling richer semantic matching be- tween queries and tabular structures. Our method outperforms state-of-the-art (SOTA) sparse, dense, and hybrid retrievers on the NQ- Tables dataset. It also...

paper website code

TABREX: Tabular Referenceless eXplainable Evaluation

Tejas Anvekar, Junha Park, Aparna Garimella, Vivek Gupta ACL 2026

Evaluating the quality of tables generated by large language models (LLMs) remains an open challenge: existing metrics either flatten ta- bles into text, ignoring structure, or rely on fixed references that limit generalization. We present TABREX, a reference-less, property- driven framework for evaluating tabular gen- eration via graph-based reasoning. TABREX converts both source text and generated tables into canonical knowledge graphs, aligns them through an LLM-guided matching process, and computes interpretable, rubric-aware scores that quantify structural and factual fidelity. The resulting metric provides controllable trade- offs between sensitivity and specificity, yield- ing human-aligned judgments and cell-level error traces. To systematically assess metric robustness, we introduce TABREX-BENCH, a large-scale benchmark spanning six domains and twelve planner-driven...

paper website code

Moneyball with LLMs: Analyzing Tabular Summarization in Sports Narratives

Ritam Upadhyay*, Naman Ahuja*, Rishabh Baral, Aparna Garimella, Vivek Gupta ACL 2026 (Findings)

Large language model (LLM) approaches to tabular summarization rely on extensive prompt engineering, decomposition pipelines, or entity- level intermediate representations to achieve strong performance. While effective, these strategies are computationally expensive and offer limited insight into how well models main- tain state over long, evolving narratives. We introduce SPORTABSET, a diagnostic bench- mark for long-context tabular summarization across two complementary sports domains that require tracking multiple entities and aggre- gating statistics under domain-specific rules. Using SporTabSet, we systematically evaluate decomposition-based strategies across several long context LLMs. Results show that...

paper website code

Decomposed Prompting Does Not Fix Knowledge Gaps, But Helps Models Say "I Don't Know"

Dhruv Madhwal*, Lyuxin David Zhang*, Dan Roth, Tomer Wolfson, Vivek Gupta ACL 2026 (Findings)

Large language models often struggle to rec- ognize their knowledge limits in closed-book question answering, leading to confident hal- lucinations. While decomposed prompting is typically used to improve accuracy, we inves- tigate its impact on reliability. We evaluate three task-equivalent prompting regimes: Di- rect, Assistive, and Incremental, across dif- ferent model scales and multi-hop QA bench- marks. We find that although accuracy gains from decomposition diminish in frontier mod- els, disagreements between prompting regimes remain highly indicative of potential errors. Be- cause factual knowledge is stable while hal- lucinations are stochastic, cross-regime agree- ment provides a precise signal of internal un- certainty. We leverage this signal...

paper

MMTabReal: Real-World Benchmark for Multimodal Table Understanding

Prasham Titiya*, Jainil Trivedi*, Chitta Baral, Vivek Gupta ACL 2026 (Findings)

Multimodal tables i.e. tabular layouts interleaved with charts, maps, icons, and color encodings are ubiquitous in real applications yet remain difficult for Multimodal Large Language Models (MLLMs). Despite advances in text and image understanding, systematic evaluation of table-centric multimodal reasoning is limited. We introduce MMTabReal, a MultiModal Table Benchmark, human-curated suite of 500 real-world tables paired with 4,021 question–answer pairs. MMTabReal spans four question types, five reasoning categories, and eight structural archetypes. Evaluations of state-of-the-art models reveal substantial gaps, especially in visual grounding, spatial alignment, and multi-step inference, with 20–40% performance drops relative to existing benchmarks. These results highlight ...

paper website code

JobMatchAI An Intelligent Job Matching Platform Using Knowledge Graphs, Semantic Search and Explainable AI

Mayank Vyas, Abhijit Chakraborty, Vivek Gupta ACL 2026 (Demo)

Recruiters and job seekers rely on search sys- tems to navigate labor markets, making candi- date matching engines critical for hiring out- comes. Most systems act as keyword filters, failing to handle skill synonyms and nonlin- ear careers, resulting in missed candidates and opaque match scores. We introduce Job- MatchAI, a production-ready system integrat- ing Transformer embeddings, skill knowledge graphs, and interpretable reranking. Our sys- tem optimizes utility across skill fit, expe- rience, location, salary, and company pref- erences, providing factor-wise explanations through resume-driven search workflows. We release JobSearch-XS benchmark and a hybrid retrieval stack combining BM25, knowledge graph and semantic components to evaluate skill generalization. We assess...

paper code

GAMED.AI: A Hierarchical Multi-Agent Framework for Automated Educational Game Generation

Shiven Agarwal* Yash Shah* Ashish Raj Shekhar* Priyanuj Bordoloi Vivek Gupta ACL 2026 (Demo)

We introduce GAMED.AI, a hierarchical multi- agent framework that transforms instructor- provided questions into fully playable, ped- agogically grounded educational games vali- dated through formal mechanic contracts. Built on phase-based LangGraph sub-graphs, deter- ministic Quality Gates, and structured Pydantic schemas, GAMED.AI supports two template families encompassing 15 interaction mechan- ics across spatial reasoning, procedural exe- cution, and higher-order Bloom’s Taxonomy objectives. Evaluated on 200 questions span- ning five subject domains, the system achieves a 90% validation pass rate against internal FOL-based structural validators...

paper website code

FD-NL2SQL: Feedback-Driven Clinical NL2SQL that Improves with Use

Suparno Roy Chowdhury*, Tejas Anvekar*, Manan Roy Choudhury*, Muhammad Ali Khan, Kaneez Zahra Rubab Khakwani, Mohamad Bassam Sonbol, Irbaz Bin Riaz, Vivek Gupta ACL 2026 (Demo)

Clinicians exploring oncology trial reposito- ries often need ad-hoc, multi-constraint queries over biomarkers, endpoints, interventions, and time, yet writing SQL requires schema ex- pertise. We demo FD-NL2SQL, a feedback- driven clinical NL2SQL assistant for SQLite- based oncology databases. Given a natural- language question, a schema-aware LLM de- composes it into predicate-level sub-questions, retrieves semantically similar expert-verified NL2SQL exemplars via sentence embeddings, and synthesizes executable SQL conditioned on the decomposition, retrieved exemplars, and schema, with post-processing validity checks. To improve with use, FD-NL2SQL incorpo- rates...

paper website code

DIAGRAMS: A Review Framework for Reasoning-Level Attribution in Diagram QA

Anirudh Iyengar Kaniyar Narayana Iyengar*, Tampu Ravi Kumar*, Manan Suri, Raviteja Bommireddy, Dinesh Manocha, Puneet Mathur*, Vivek Gupta* ACL 2026 (Demo)

Diagram question answering (Diagram QA) requires reasoning-level attribution that links each question–answer pair to all visual re- gions needed to derive the answer, rather than only the region containing the final response. Creating such structured evidence across dia- grams, charts, maps, circuits, and infograph- ics is time-consuming, and existing annotation tools tightly couple their interfaces to dataset- specific formats. We present DIAGRAMS, a lightweight, schema-driven review framework that decouples interface logic from dataset- specific JSON structures through an internal meta-schema and dataset adapters. Given an image and QA pair with optional candidate re- gions, the system performs...

paper website code

ViTaB-A: Evaluating Multimodal Large Language Models on Visual Table Attribution

Yahia Alqurnawi*, Preetom Biswas*, Anmol Rao*, Tejas Anvekar, Chitta Baral, Vivek Gupta ES-Reasoning @ ICLR 2026

Multimodal Large Language Models (mLLMs) are often used to answer ques- tions in structured data such as tables in Markdown, JSON, and images. While these models can often give correct answers, users also need to know where those answers come from. In this work, we study structured data attribution/citation, which is the ability of the models to point to the specific rows and columns that support an answer. We evaluate several mLLMs across different table formats and prompting strategies. Our results show a clear gap between question answering and evidence attribution. Although question answering accuracy remains moder- ate...

paper website code

Self-Aug: Query and Entropy Adaptive Decoding for Large Vision-Language Models

ICLR 2026

Large Vision-Language Models (LVLMs) have demonstrated remarkable multi-modal capabilities, but they inherit the tendency to hallucinate from their underlying language models. While visual contrastive decoding has been proposed to mitigate this issue, existing methods often apply generic visual augmentations that disre- gard the specific context provided by the text query, limiting their effectiveness. This study introduces a novel training-free decoding strategy that addresses these limitations, featuring two key contributions. First, a self-augmentation prompting strategy that leverages the intrinsic knowledge of the model to dynamically align semantics between the query and the visual augmentation. Second, an adaptive thresholding algorithm that adaptively adjusts next token candidate size based on the output sparsity, utilizing full information from the logit distribution. Ex- tensive experiments...

paper website code

Harnessing Meta-Learning for Controllable Full-Frame Video Stabilization

Muhammad Kashif Ali, Eun Woo Im, Dongjin Kim, Tae Hyun Kim, Vivek Gupta, Haonan Luo, Tianrui Li IEEE TPAMI 2026

Video stabilization remains a fundamental problem in computer vision, particularly pixel-level synthesis solutions for video stabilization, which synthesize full-frame outputs, add to the complexity of this task. These methods aim to enhance stability while synthesizing full-frame videos, but the inherent diversity in motion profiles and visual content present in each video sequence makes robust generalization with fixed parameters difficult. To address this, we present a novel method that improves pixel-level synthesis video stabilization methods by rapidly adapting models to each input video at test time. The proposed approach takes advantage of low-level visual cues available during inference to improve both the stability and visual quality of the output. Notably...

paper

SCoPE: Planning for Hybrid Querying over Clinical Trial Data

Suparno Roy Chowdhury, Manan Roy Choudhury, Tejas Anvekar, Muhammad Ali Khan, Kaneez Zahra Rubab Khakwani, Mohamad Bassam Sonbol, Irbaz Bin Riaz, Vivek Gupta BioNLP 2026

SCoPE studies clinical trial table reasoning, where answers must be inferred from row evidence through semantic normalization, classification, extraction, and lightweight domain reasoning. The planner-executor framework decomposes row selection, structured planning, and grounded execution, improving row-level reasoning over direct prompting and tabular baselines on 1,500 oncology clinical-trial questions.

website code

MAPVERSE: A Benchmark for Geospatial Question Answering on Diverse Real-World Maps

Sharat Bhat, Harshita Khandelwal, Tushar Kataria, Vivek Gupta WACV 2026

Maps are powerful carriers of structured and contextual knowledge, encompassing geography, demographics, in- frastructure, and environmental patterns. Reasoning over such knowledge requires models to integrate spatial re- lationships, visual cues, real-world context, and domain- specific expertise-capabilities that current large language models (LLMs) and vision–language models (VLMs) still struggle to exhibit consistently. Yet, datasets used to bench- mark VLMs on map-based reasoning remain narrow in scope, restricted to specific domains, and heavily reliant on artificially generated content (outputs from LLMs or pipeline-based methods), offering limited depth for evaluat- ing genuine geospatial reasoning. To address this gap, we present MAPVERSE, a large-scale benchmark built on real- world maps. It comprises...

paper

The Perceptual Observatory Characterizing Robustness and Grounding in MLLMs

Tejas Anvekar, Fenil Bardoliya, Pavan Turaga, Chitta Baral, Vivek Gupta WACV 2026

We present The Perceptual Observatory, a framework that characterizes MLLMs across verticals like: (i) simple vision tasks, such as face matching and text-in-vision comprehension capabilities; (ii) local-to-global understanding, encompassing image matching, grid pointing game, and attribute localization, which tests general visual grounding. Each vertical is instantiated with ground-truth datasets of faces and words,systematically perturbed through pixel-based augmentations and diffusion-based stylized illusions. The Perceptual Observatory moves beyond leaderboard accuracy to...

paper website code

Integrity Shield: A System for Ethical AI Use & Authorship Transparency in Assessments

Ashish Raj Shekhar, Shiven Agarwal, Priyanuj Bordoloi, Yash Shah, Tejas Anvekar, Vivek Gupta EACL 2026 (Demo)

Large Language Models (LLMs) can now solve entire exams directly from uploaded PDF as- sessments, raising urgent concerns about aca- demic integrity and the reliability of grades and credentials. Existing watermarking tech- niques either operate at the token level or assume control over the model’s decoding process, making them ineffective when stu- dents query proprietary black-box systems with instructor-provided documents. We present IN- TEGRITY SHIELD, a document-layer wa- termarking system that embeds schema-aware, item-level watermarks into assessment PDFs while keeping their human-visible appearance unchanged. These watermarks consistently pre- vent MLLMs from answering shielded exam PDFs and encode stable, item-level signatures that can be reliably recovered from model or student responses. Across 30 exams...

paper website code

No Universal Prompt: Unifying Reasoning through Adaptive Prompting for Temporal Table Reasoning

Abhishek Rajgaria,Kushagra Dixit, Mayank Vyas, Harshavardhan Kalalbandi, Dan Roth, Vivek Gupta AACL 2025

Temporal Table Reasoning poses a significant challenge for Large Language Models (LLMs), requiring effective reasoning to extract relevant insights. Despite existence of multiple prompting methods, their impact on table reasoning remains largely unexplored. Furthermore, model performance varies drastically across different table and context structures, making it difficult to determine an optimal approach. This work investigates multiple prompting technique on diverse table types to determine that performance depends on factors such as entity type, table structure, requirement of additional context and question complexity, with "NO" single method consistently outperforming others. To address this, we introduce SEAR, an adaptive prompting framework inspired by human reasoning that dynamically adjusts to context and integrates structured reasoning. SEAR_Unified, its cost-efficient variant. We also demonstrate that optional table refactoring (preprocessing) enhances both approaches when tables lack structural consistency. Our results demonstrate that SEAR prompts achieve superior performance across all table types compared to baseline prompting techniques...

paper website data

Rethinking Information Synthesis in Multimodal Question Answering A Multi-Agent Perspective

Tejas Anvekar, Krishna Singh Rajput, Chitta Baral, Vivek Gupta AACL 2025

Recent advances in multimodal question answering have primarily focused on combining heterogeneous modalities or fine-tuning multimodal large language models. While these approaches have shown strong performance, they often rely on a single, generalized reasoning strategy, overlooking the unique characteristics of each modality ultimately limiting both accuracy and interpretability. To address these limitations, we propose MAMMQA , a multi-agent QA framework for multimodal inputs spanning text, tables, and images. Our system includes...

paper website code slides

SPORTSQL: An Interactive System for Real-Time Sports Reasoning and Visualization

Sebastian Martinez, Naman Ahuja, Fenil Bardoliya, Chris Bryan, Vivek Gupta AACL 2025

We present a modular, interactive system SPORTSQL for natural language querying and visualization of dynamic sports data, with a focus on the English Premier League (EPL). The system translates user questions into executable SQL over a live, temporally indexed database constructed from real-time Fantasy Premier League (FPL) data. It supports both tabular and visual outputs, leveraging symbolic reasoning capabilities of Large Language Models (LLMs) for query parsing, schema linking, and visualization selection. To evaluate...

paper website code

InterChart: Benchmarking Visual Reasoning Across Decomposed and Distributed Chart Information

Anirudh Iyengar Kaniyar Narayana Iyengar, Srija Mukhopadhyay, Adnan Qidwai, Shubhankar Singh, Dan Roth, Vivek Gupta AACL 2025

We introduce InterChart, a diagnostic benchmark that evaluates how well vision-language models (VLMs) reason across multiple related charts, a task central to real-world applications such as scientific reporting, financial analysis, and public policy dashboards. Unlike prior benchmarks focusing on isolated, visually uniform charts, InterChart challenges models with diverse question types ranging from entity inference and trend correlation to numerical estimation and abstract multi-step reasoning grounded in 2-3 thematically or structurally related charts. We organize the benchmark into three tiers of increasing difficulty: (1) factual reasoning over individual charts, (2) integrative analysis across synthetically aligned chart sets, and (3) semantic inference over visually complex, real-world chart pairs. Our evaluation...

paper website code data slides

Follow the Flow: Fine-grained Flowchart Attribution with Neurosymbolic Agents

Manan Suri, Puneet Mathur, Nedim Lipka, Franck Dernoncourt, Ryan A. Rossi, Vivek Gupta, Dinesh Manocha EMNLP 2025

Flowcharts are a critical tool for visualizing decision-making processes. However, their non-linear structure and complex visual-textual relationships make it challenging to interpret them using LLMs, as vision-language models frequently hallucinate nonexistent connections and decision paths when analyzing these diagrams. This leads to compromised reliability for automated flowchart processing in critical domains such as logistics, health, and engineering. We introduce the task of Fine-grained Flowchart Attribution, which traces specific components grounding a flowchart referring LLM response. Flowchart Attribution ensures the verifiability of LLM predictions and improves explainability by linking generated responses to the flowchart's structure. We propose FlowPathAgent, a neurosymbolic agent that performs fine-grained post hoc attribution through graph-based reasoning. It first segments...

paper code slides

Weaver: Interweaving SQL and LLM for Table Reasoning

Rohit Khoja, Devanshu Gupta, Yanjie Fu, Dan Roth, Vivek Gupta EMNLP 2025

Querying tables with unstructured data is challenging due to the presence of text (or image), either embedded in the table or in external paragraphs, which traditional SQL struggles to process, especially for tasks requiring semantic reasoning. While Large Language Models (LLMs) excel at understanding context, they face limitations with long input sequences. Existing approaches that combine SQL and LLMs typically rely on rigid, predefined work flows, limiting their adaptability to complex queries. To address these issues, we introduce Weaver , a modular pipeline that dynamically integrates SQL and LLMs for table-based question answering (TableQA). Weaver generates...

paper website code slides

TABARD: A Novel Benchmark for Tabular Anomaly Analysis, Reasoning and Detection

Manan Roy Choudhury, Anirudh Iyengar, Shikhhar Siingh, Sugeeth Puranam, Vivek Gupta EMNLP 2025

We study the capabilities of large language models (LLMs) in detecting fine-grained anomalies in tabular data. Specifically, we examine: (1) how well LLMs can identify diverse anomaly types—including factual, logical, temporal, and value-based errors; (2) the impact of prompt design and prompting strategies; and (3) the effect of table structure and anomaly type on detection accuracy. To this end, we introduce TABARD, a new benchmark constructed by perturbing tables from WikiTQ, FeTaQA, Spider, and BEAVER. The dataset spans multiple domains and eight anomaly categories, including paired clean and corrupted tables. We evaluate LLMs using direct, indirect, and Chain-of-Thought (CoT) prompting. Our results reveal notable limitations in standard prompting, especially for complex reasoning tasks and longer tables. To overcome these issues, we propose a unified framework combining multi-step prompting, self-verification, and constraint-based rule execution. Our approach...

paper website data code slides

Federated Retrieval-Augmented Generation: A Systematic Mapping Study

Abhijit Chakraborty, Chahana Dahal, Vivek Gupta EMNLP 2025 (Findings)

Federated Retrieval-Augmented Generation (Federated RAG) combines Federated Learning (FL),which enables distributed model training without exposing raw data, with Retrieval-Augmented Generation (RAG), which improves the factual accuracy of language models by grounding outputs in external knowledge. As large language models are increasingly deployed in privacy-sensitive domains such as healthcare, finance, and personalized assistance, Federated RAG offers a promising framework for secure, knowledge-intensive natural language processing (NLP). To the best of our knowledge, this paper presents the first systematic mapping study of Federated RAG, covering literature published between 2020 and 2025. Following Kitchenham’s...

paper

M-HELP: Using Social Media Data to Detect Mental Health Help-Seeking Signals

MSVPJ Sathvik, Zuhair Hasan Shaik, Vivek Gupta EMNLP 2025 (Findings)

Mental health disorders are a global crisis. While various datasets exist for detecting such disorders, there remains a critical gap in identifying individuals actively seeking help. This paper introduces a novel dataset, M-HELP, specifically designed to detect help-seeking behavior on social media. The dataset goes beyond traditional labels by identifying not only help-seeking activity but also specific mental health disorders and their underlying causes, such as relationship challenges or financial stressors. AI models...

paper data

MapIQ: Benchmarking Multimodal Large Language Models for Map Question Answering

V Srivastava, F Lei, S Mukhopadhyay, V Gupta, R Maciejewski COLM 2025

Recent advancements in multimodal large language models (MLLMs) have driven researchers to explore how well these models read data visualizations, e.g., bar charts, scatter plots. More recently, attention has shifted to visual question answering with maps (Map-VQA). However, Map-VQA research has primarily focused on choropleth maps, which cover only a limited range of thematic categories and visual analytical tasks. To address these gaps, we introduce MapIQ, a benchmark dataset comprising 14,706 question-answer pairs across three map types—choropleth maps, cartograms, and proportional symbol maps spanning topics from six distinct themes (e.g., housing, crime). We evaluate multiple MLLMs using six visual analytical tasks, comparing their performance against one another and a human baseline. An additional experiment...

paper

TabXEval: Why this is a Bad Table? An eXhaustive Rubric for Table Evaluation

Vihang Pancholi, Jainit Sushil Bafna, Tejas Anvekar, Manish Shrivastava, Vivek Gupta ACL 2025

Evaluating tables qualitatively and quantitatively poses a significant challenge, as standard metrics often overlook subtle structural and content-level discrepancies. To address this, we propose a rubric-based evaluation frame work that integrates multi-level structural descriptors with fine-grained contextual signals, enabling more precise and consistent table comparison. Building on this, we introduce TabXEval, an eXhaustive and eXplainable two-phase evaluation framework. TabXEval first aligns reference and predicted...

paper website code slides

Map&Make: Schema Guided Text to Table Generation

Naman Ahuja, Fenil Bardoliya, Chitta Baral, Vivek Gupta ACL 2025

Transforming dense, unstructured text into interpretable tables—commonly referred to as Text-to-Table generation—is a key task in information extraction. Existing methods often overlook what complex information to extract and how to infer it from text. We present Map&Make, a versatile approach that decomposes text into atomic propositions to infer latent schemas, which are then used to generate tables capturing both qualitative nuances and quantitative facts. We evaluate...

paper website code slides

PRAISE: Enhancing Product Descriptions with LLM-Driven Structured Insights

Adnan Qidwai, Srija Mukhopadhyay, Prerana Khatiwada, Dan Roth, Vivek Gupta ACL 2025(Demo)

Accurate and complete product descriptions are crucial for e-commerce, yet seller-provided information often falls short. Customer reviews offer valuable details but are laborious to sift through manually. We present PRAISE: Product Review Attribute Insight Structuring Engine, a novel system that uses Large Language Models (LLMs) to automatically extract, compare, and structure insights from customer reviews and seller descriptions. PRAISE provides users with an intuitive interface to identify missing, contradictory, or partially matching details between these two sources, presenting the discrepancies in a clear, structured format alongside supporting evidence from reviews. This allows...

paper website code

LLM-Symbolic Integration for Robust Temporal Tabular Reasoning

Atharv Kulkarni, Kushagra Dixit, Vivek Srikumar, Dan Roth, Vivek Gupta ACL 2025

Temporal tabular question answering presents a significant challenge for Large Language Models (LLMs), requiring robust reasoning over structured data—a task where traditional prompting methods often fall short. These methods face challenges such as memorization, sensitivity to table size, and reduced performance on complex queries. To overcome these limitations, we introduce TEMPTABQA-C, a synthetic dataset designed for systematic and controlled evaluations, alongside a symbolic intermediate representation that transforms tables into database schemas. This structured approach allows LLMs to generate and execute SQL queries, enhancing generalization and mitigating biases. By incorporating...

paper website code data slides

GETReason: Enhancing Image Context Extraction through Hierarchical Multi-Agent Reasoning

Shikhhar Siingh, Abhinav Rawat, Chitta Baral, Vivek Gupta ACL 2025

Publicly significant images from events carry valuable contextual information with applications in domains such as journalism and education. However, existing methodologies of ten struggle to accurately extract this contextual relevance from images. To address this challenge, we introduce GETREASON(Geospatial Event Temporal Reasoning), a framework designed to go beyond surface level image descriptions and infer deeper con textual meaning. We hypothesize that extracting global event, temporal, and geospatial information from an image enables a more accurate understanding of its contextual significance. We also introduce a new metric GREAT (Geospatial, Reasoning and Event Accuracy with Temporal alignment) for a reasoning capturing evaluation. Our layered multi-agentic approach, evaluated...

paper website code slides

MAPWise: Evaluating Vision-Language Models for Advanced Map Queries

Srija Mukhopadhyay, Abhishek Rajgaria, Prerana Khatiwada, Manish Shrivastava, Dan Roth, Vivek Gupta NAACL 2025

Vision-language models (VLMs) excel at tasks requiring joint understanding of visual and linguistic information. A particularly promising yet under-explored application for these models lies in answering questions based on various kinds of maps. This study investigates the efficacy of VLMs in answering questions based on choropleth maps, which are widely used for data analysis and representation. To facilitate and encourage research in this area, we introduce a novel map-based question-answering benchmark, consisting of maps from three geographical regions (United States, India, China), each containing 1000 questions. Our benchmark...

paper website code data slides

Leveraging LLM for Synchronizing Information Across Multilingual Tables

Siddharth Khincha, Tushar Kataria, Ankita Anand, Dan Roth, Vivek Gupta NAACL 2025

The vast amount of online information today poses challenges for non-English speakers, as much of it is concentrated in high-resource languages such as English and French. Wikipedia reflects this imbalance, with content in low-resource languages frequently outdated or incomplete. Recent research has sought to improve cross-language synchronization of Wikipedia tables using rule-based methods. These approaches can be effective, but they struggle with complexity and generalization. This paper explores large language models (LLMs) for multilingual information synchronization, using zero-shot prompting as a scalable solution. We introduce the Information Updation dataset, simulating the real-world process of updating outdated Wikipedia tables, and evaluate LLM performance. Our findings...

paper website slides

TRANSIENT TABLES: Evaluating LLMs’ Reasoning on Temporally Evolving Semi-structured Tables

Abhilash Shankarampeta, Harsh Mahajan, Tushar Kataria, Dan Roth, Vivek Gupta NAACL 2025

Humans continuously make new discoveries, and understanding temporal sequence of events leading to these breakthroughs is essential for advancing science and society. This ability to reason over time allows us to identify future steps and understand the effects of financial and political decisions on our lives. However, large language models (LLMs) are typically trained on static datasets, limiting their ability to perform effective temporal reasoning. To assess the temporal reasoning capabilities of LLMs, we present the TRANSIENTTABLES dataset, which comprises 3,971 questions derived from over 14,000 tables, spanning 1,238 entities across multiple time periods. We introduce a template-based question-generation pipeline that harnesses LLMs to refine both templates and questions. Additionally, we establish baseline...

paper website code slides

H-STAR: LLM-driven Hybrid SQL-Text Adaptive Reasoning on Tables

Nikhil Abhyankar, Vivek Gupta, Dan Roth, Chandan Reddy NAACL 2025

Tabular reasoning involves interpreting natural language queries about tabular data, which presents a unique challenge of combining language understanding with structured data analysis. Existing methods employ either textual reasoning, which excels in semantic interpretation but struggles with mathematical operations, or symbolic reasoning, which handles computations well but lacks semantic understanding. This paper introduces a novel algorithm H-STAR that integrates both symbolic and semantic (textual) approaches in a two-stage process to address these limitations. H-STAR employs: (1) step-wise table extraction using ‘multi-view’ column retrieval followed by row extraction, and (2) adaptive reasoning that adapts reasoning strategies based on question types, utilizing semantic reasoning for direct lookup and complex lexical queries while augmenting textual reasoning with symbolic reasoning support for quantitative and logical tasks. Our extensive...

paper website code slides

Enhancing Temporal Understanding in LLMs for Semi-structured Tables

Irwin Deng, Kushagra Dixit, Dan Roth, Vivek Gupta NAACL 2025 (Findings)

Temporal reasoning over tabular data presents substantial challenges for large language models (LLMs), as evidenced by recent research. In this study, we conduct a comprehensive analysis of temporal datasets to pinpoint the specific limitations of LLMs. Our investigation leads to enhancements in TempTabQA, a benchmark specifically designed for tabular temporal question answering. We provide critical insights for enhancing LLM performance in temporal reasoning tasks with tabular data. Furthermore, we introduce a novel approach, C.L.E.A.R to strengthen LLM capabilities in this domain. Our findings demonstrate that our method im proves evidence-based reasoning across various models. Additionally, our experimental...

paper website code

NTSEBENCH: Cognitive Reasoning Benchmark for Vision Language Models

Pranshu Pandya, Vatsal Gupta, Agney S Talwarr, Tushar Kataria, Dan Roth, Vivek Gupta NAACL 2025

Cognitive textual and visual reasoning tasks, including puzzles, series, and analogies, demand the ability to quickly reason, decipher, and evaluate patterns both textually and spatially. Due to extensive training on vast amounts of human-curated data, large language models (LLMs) and vision language models (VLMs) excel in common-sense reasoning tasks, but still struggle with more complex reasoning that demands deeper cognitive understanding. We introduce NTSEBENCH, a new dataset designed to evaluate cognitive multimodal reasoning and problem-solving skills of large models. The dataset contains 2,728 multiple-choice questions, accompanied by a total of 4,642 images, spanning 26 categories....

paper website code data slides

Evaluating Concurrent Robustness of Language Models Across Diverse Challenge Sets

Vatsal Gupta, Pranshu Pandya, Tushar Kataria, Vivek Gupta, Dan Roth
EMNLP 2024

Language models, characterized by their black box nature, often hallucinate and display sensitivity to input perturbations, causing concerns about trust. To enhance trust, it is imperative to gain a comprehensive understanding of the model’s failure modes and develop effective strategies to improve their performance. In this study, we introduce a methodology designed to examine how input perturbations affect language models across various scales, including pre-trained models and large language models (LLMs). Utilizing fine-tuning, we enhance the model’s robustness to input perturbations. Additionally, ...

paper website code slides

Unraveling the Truth: Do VLMs really Understand Charts? A Deep Dive into Consistency and Robustness

Srija Mukhopadhyay, Adnan Qidwai, Aparna Garimella, Pritika Ramu, Vivek Gupta, Dan Roth
EMNLP 2024 (Findings)

Chart question answering (CQA) is a crucial area of Visual Language Understanding. However, the robustness and consistency of current Visual Language Models (VLMs) in this field remain under-explored. This paper evaluates state-of-the-art VLMs on comprehensive datasets, developed specifically for this study, encompassing diverse question categories and chart formats. We investigate two key aspects: 1) the models' ability to handle varying levels of chart and question complexity, and 2) their robustness across different visual representations of the same underlying data. Our analysis reveals significant performance variations based on question and chart types, highlighting both strengths and weaknesses of current models. Additionally,...

paper website code

Knowledge-Aware Reasoning over Multimodal Semi-structured Tables

Suyash Vardhan Mathur, Jainit Sushil Bafna, Kunal Kartik, Harshita Khandelwal, Manish Shrivastava, Vivek Gupta, Mohit Bansal, Dan Roth EMNLP 2024 (Findings)

Existing datasets for tabular question answering typically focus exclusively on text within cells. However, real-world data is inherently multimodal, often blending images such as symbols, faces, icons, patterns, and charts with textual content in tables. With the evolution of AI models capable of multimodal reasoning, it is pertinent to assess their efficacy in handling such structured data. This study investigates whether current AI models can perform knowledge-aware reasoning on multimodal structured data. We explore their ability to reason on tables that integrate both images and text, introducing MMTABQA, a new dataset designed for this purpose. Our experiments...

paper website code data slides

FlowVQA: Mapping Multimodal Logic in Visual Question Answering with Flowcharts

Shubhankar Singh, Purvi Chaurasia, Yerram Varun, Pranshu Pandya, Vatsal Gupta, Vivek Gupta, Dan Roth ACL 2024 (Findings)

Existing benchmarks for visual question answering lack in visual grounding and complexity, particularly in evaluating spatial reasoning skills. We introduce FlowVQA, a novel benchmark aimed at assessing the capabilities of visual question-answering multimodal language models in reasoning with flowcharts as visual contexts. FlowVQA comprises 2,272 carefully generated and human-verified flowchart images from three distinct content sources, along with 22,413 diverse question-answer pairs, to test a spectrum of reasoning tasks, including information localization, decision-making, and logical progression. We conduct a thorough baseline evaluation on a suite of both open-source and proprietary multimodal language models using various strategies, followed by an analysis of directional bias. The results...

paper website code data slides

Evaluating LLMs’ Mathematical Reasoning in Financial Document Question Answering

Pragya Srivastava, Manuj Malik, Vivek Gupta, Tanuja Ganu, Dan Roth ACL 2024 (Findings)

Large Language Models (LLMs), excel in natural language understanding, but their capability for complex mathematical reasoning with a hybrid of structured tables and unstructured text remain uncertain. This study explores LLMs’ mathematical reasoning on four financial tabular question-answering datasets: TATQA, FinQA, ConvFinQA, and Multihiertt. Through extensive experiments with various models and prompting techniques, we assess how LLMs adapt to complex tables and mathematical tasks. We focus on sensitivity to table complexity and performance variations with an increasing number of arithmetic reasoning steps. The results...

paper slides

ChartCheck: An Evidence-Based Fact-Checking Dataset over Real-World Chart Images

Mubashara Akhtar, Nikesh Subedi, Vivek Gupta, Sahar Tahmasebi, Oana Cocarascu, Elena Simperl ACL 2024 (Findings)

Whilst fact verification has attracted substantial interest in the natural language processing community, verifying misinforming statements against data visualizations such as charts has so far been overlooked. Charts are commonly used in the real-world to summarize and communicate key information, but they can also be easily misused to spread misinformation and promote certain agendas. In this paper, we introduce ChartCheck, a novel, large-scale dataset for explainable fact-checking against Chart: Evidence real-world charts, consisting of 1.7k charts and 10.5k human-written claims and explanations. We systematically...

paper code

Enhancing Question Answering on Charts Through Effective Pre-training Tasks

Ashim Gupta, Vivek Gupta, Shuo Zhang, Yujie He, Ning Zhang, Shalin Shah BlackboxNLP 2024 (Findings)

To completely understand a document, the use of textual information is not enough. Under standing visual cues, such as layouts and charts, is also required. While the current state-of the-art approaches for document understanding (both OCR-based and OCR-free) work well, we have not found any other works conducting a thorough analysis of their capabilities and limitations. Therefore, in this work, we address the limitation of current VisualQA models when applied to charts and plots. To investigate shortcomings of the state-of-the-art models, we conduct a comprehensive behavioral analysis, using ChartQA as a case study. Our findings ...

paper

RECENT PUBLICATIONS

Complex Data (Structured Data)

PAST PUBLICATIONS