DocLayout-VL: A Foundational Model for Hierarchical, Open-Set and Promptable Document Layout Segmentation

Reimagining Document Layout Understanding

Document layout understanding has been stuck in a pre-foundation-model era: fixed ontologies, non-hierarchical labels, and axis-aligned boxes. That abstraction breaks exactly where real documents become useful: irregular regions, nested sections, domain-specific labels, curved text, and interactive user intent.

DocLayout-VL resets the formulation. It treats layout as contextualized, hierarchical, open-set, and promptable segmentation. For a page, the model predicts a coherent layout tree, assigns open-vocabulary semantic labels, grounds every node with a pixel-accurate mask, and can adapt the parse at inference time through natural language prompts.

At the center is the Hierarchical Grounding Contextualizer, which uses tree-aware attention over parent, child, and sibling relationships before mask decoding. Trained on HOPLay-Docs-80M and evaluated on HOPLay-Docs-Bench, DocLayout-VL moves layout understanding from static preprocessing to a controllable, semantically grounded interface for document intelligence.

Core idea: if LLM-era document systems need layout as a reasoning substrate, then layout parsing cannot remain a fixed-schema box detector. It needs to be open-set, hierarchical, mask-grounded, and promptable.

Foundational Layout Capabilities

Task formulation. Agentic document workflows need more than OCR and rectangles. They need a page representation that can be queried, revised, grounded, and composed across downstream reasoning steps.

Layout as Segmentation

Agents cannot reliably operate on coarse bounding boxes when documents contain curved text, overlapping regions, dense tables, or irregular graphics. Pixel-accurate masks give every extracted region a precise visual footprint, making redaction, retrieval chunking, citation, and verification spatially trustworthy.

Open-Set Layout Labels

Real workflows do not share one universal ontology. A financial report, medical form, slide deck, and government circular all demand different concepts. Open-set labeling lets the model infer domain-specific layout semantics instead of forcing every page into a small closed taxonomy.

Hierarchical Structure

Document meaning is nested: sections contain headings, tables contain cells, forms contain fields, and panels contain subregions. A layout tree exposes these parent-child-sibling relations so agents can reason over context, reading order, containment, and multi-step extraction plans.

Promptable Control

Agentic systems need to ask for exactly the layout they need: redact only PII, extract all invoice fields, isolate figure captions, or focus on a spatial region. Promptable layout understanding turns parsing into an interactive interface rather than a one-shot preprocessing artifact.

Together, these four capabilities make layout a reusable reasoning substrate: open vocabulary for new domains, hierarchy for structure, masks for evidence, and prompts for control.

Agentic Document Workflows Mask-Grounded Parsing Open-Vocabulary Layout Hierarchical Segmentation Promptable Control

Interactive Demo

Watch DocLayout-VL generate a structured layout tree in real time. Each [LAY] token is linked to a pixel-accurate mask that is revealed on the document image as the model streams its output.

DocLayout-VL

Method

DocLayout-VL architecture. A vision-language model generates a serialized XML layout tree with special [LAY] tokens. The Hierarchical Grounding Contextualizer refines those tokens using tree-aware attention, and a segmentation module decodes pixel-accurate masks for each layout region.

Layout Tree Generation

The VLM autoregressively produces a semantic XML-style layout tree, creating an editable structured parse rather than a flat list of detections.

Hierarchical Grounding Contextualizer

HGC refines [LAY] token embeddings with tree-aware attention over parent, child, and sibling relations, enforcing structural locality before mask prediction.

Mask-Level Grounding

A segmentation module converts the contextualized layout tokens into masks, grounding every semantic node in precise page evidence.

Prefer to see it in motion?

Step through the full pipeline — document + prompt → VLM → Hierarchical Grounding Contextualizer (with its tree-aware attention) → Segmentation Module → masks, answer, and layout tree — in an animated, camera-guided walkthrough.

Open fullscreen

HOPLay-Docs-80M Dataset & Benchmark

HOPLay-Docs-80M supports foundation-scale learning for flat, hierarchical, and prompt-conditioned layout segmentation. HOPLay-Docs-Bench is a human-verified suite for evaluating open-set, hierarchical, curved, and promptable layout understanding.

2.5M

Pages

80M

Pixel Masks

5M

Promptable Annotations

400K

Open-Set Labels

58

Document Types

FlatLay-Docs-25M

Open-set flat layout grounding with 25M regions and 250K unique labels.

HierLay-Docs-40M

Nested layout structure learning with 40M regions and hierarchy-aware supervision.

PromptLay-Docs-15M

Instruction data for prompt-conditioned extraction, with 15M regions and 5M annotations.

FlatLay-Docs-Bench HierLay-Docs-Bench CurveLay-Docs-Bench PromptLay-Docs-Bench 10 Tasks

Benchmark Results

DocLayout-VL turns the hardest layout capabilities into a single strong result: open-set semantics, hierarchy, curved regions, promptable control, and closed-set transfer without task-specific detectors. Select a benchmark and metric to explore every reported model comparison.

79.4

Overall Score
DocLayout-VL-8B-it

86.7

Open mIoU
Flat Benchmark

77.5

Open mIoU
Hierarchical Benchmark

87.9

Open mIoU
Curved Benchmark

87.2

G-Eval
Promptable Benchmark

Foundational Layout Understanding

Flat, hierarchical, curved, promptable, and overall evaluation on HOPLay-Docs-Bench.

Zero-Shot Closed-Set Layout Detection

mAP@50:95 across ten established document layout datasets.