DIAGRAMS: A Review Framework for Reasoning-Level Attribution in Diagram QA

Overview

DIAGRAMS is a review framework for Diagram Question Answering (Diagram QA) that introduces reasoning-level attribution. Instead of only checking whether a final answer matches ground truth, the framework evaluates whether the model's reasoning is grounded in the right sequence of diagram regions.

DIAGRAMS decouples UI logic from dataset-specific JSON using an internal meta-schema and dataset adapters. In a review-first workflow, model suggestions are verified by humans instead of drawing all regions from scratch.

How to use?

Follow this workflow to use the DIAGRAMS tool for annotation and review.

Load

Load a dataset record including the image, question-answer pair, and candidate evidence regions into the tool.
Inspect

Inspect the model-proposed evidence regions and verify whether each region is relevant to the selected QA pair.
Modify

Edit, add, or remove regions as needed to create a correct and complete reasoning path.
Generate

If QA pairs or bounding boxes are missing, use generation options to create them with the multimodal model.
Verify

Verify the final reasoning chain from supporting regions to the final answer region before finalizing.
Export

Export finalized annotations as structured JSON files for downstream training and evaluation.