ECCV 2026

DocLayout-VL

A Foundational Model for Hierarchical, Open-Set and Promptable Document Layout Segmentation

Venkata Kesav Venna* · Srihari Bandarupalli* · Anirudh Srinivasan* · Raghuveer R · Sai Madhusudan Gunda · Ravi Kiran Sarvadevabhatla
* equal contribution  ·  BharatGen · IIIT Hyderabad  ·  doclayoutvl.github.io

DocLayout-VL · The leap

From grounding answers… to grounding the entire page

Document layout analysis today:

fixed classes

every page forced into 5–74 predefined classes

flat layout detection

no hierarchy — sections, headings, text all peers

box detection

axis-aligned boxes misrepresent curved text, wrapped columns, circular charts

The reframe: layout analysis as contextualized, hierarchical, open-set, promptable segmentation — one model, one queryable interface.
DocLayout-VL teaser
DocLayout-VLMotivation
DocLayout-VL · Task

fθ(d, P) → (Tˆ, Mˆ, Xˆ)  page + prompt → tree, masks, optional explanations

Task A: layout as segmentation

A · masks

every region gets a pixel-accurate mask — even a donut-shaped magazine spread

Task B: open-set labels

B · open-set labels

the same policy parsed with generic classes (B1) or insurance-specific ones — "premium breakdown", "authorization signature" (B2). The prompt defines the vocabulary.

DocLayout-VLTasks A · B
DocLayout-VL · Task

Task C — the page as a hierarchy tree

Segmented newspaper front page
Hierarchical layout tree

"Give the full hierarchical layout parse for this newspaper" → a layout tree Tˆ = (Vˆ, Eˆ): teaser-strip, masthead and main-grid, down to headlines, media units and image captions.

DocLayout-VLTask C · hierarchy
DocLayout-VL · Task

Task D — promptable parsing

Task D: promptable parsing examples
(D1) "Mask all personal identifiable information of the passport holder" → photo, signature, MRZ block… · (D2) "Find the figure to the left of a table" → a spatial query answered with a grounded region + explanation.
DocLayout-VLTask D · promptable
DocLayout-VL · Output representation

A serialized layout tree, interleaved with [LAY] tokens

<section>
  heading[LAY]
  paragraph[LAY]
  paragraph[LAY]
</section>[LAY]

The generated sequence directly encodes the hierarchy: leaves as label[LAY], parent nodes closed by </tag>[LAY]. Each [LAY] token corresponds one-to-one with a spatial region; labels come from the prompt-defined vocabulary.

sec head para para one [LAY] per node → one mask per node
DocLayout-VLSerialization
DocLayout-VL · Architecture

Architecture

DocLayout-VL architecture
DocLayout-VLArchitecture
DocLayout-VL · Interactive

DocLayout-VL in action

Click inside to drive it · Space pause · R restart · F fit — then use ‹ › to continue the talk open standalone ↗
DocLayout-VLInteractive architecture
DocLayout-VL · Training objective

The loss function

𝓛total =
𝓛lmcross-entropy
+
𝓛segDice + BCE per mask
+
𝓛bgbackground exclusion
+
𝓛structcontainment + non-overlap
parent child must stay inside

Containment

a child's mask is penalized wherever it leaves its parent's

siblings must not overlap

Sibling non-overlap

siblings can't claim the same pixels

𝓛bg — no mask may spill into whitespace outside the page's ground-truth foreground.

DocLayout-VLLosses
DocLayout-VL · Training strategy

A three-stage curriculum

1 · flat layouts dense HGC attention ❄ VLM vision encoder frozen 2 · hierarchical switch to tree-sparse attention learn structure 3 · instruction tuning promptability ❄ SAM vision + prompt encoders train LLM + mask decoder
DocLayout-VLCurriculum
DocLayout-VL · Data & benchmark

HOPLay-Docs-80M — layout supervision at foundation scale

2.5M
pages
80M
masks
400K
open-set labels
58
doc types · 12 domains
40
languages, strong Indic coverage

Engine: VLM page reconstruction → detector relabeling → SAM3 pixel masks → Set-of-Mark instruction data → VLM + LLM verification.

HOPLay-Docs-Bench: 5,000 docs · 51,241 regions, human-verified · four tracks (flat / hierarchical / curved / promptable) · OS = mean of track averages.

Splits mirror the curriculum: FlatLay-Docs-25M · HierLay-Docs-40M · PromptLay-Docs-15M (5M instruction annotations)

DocLayout-VLDataset & benchmark
DocLayout-VL · Results

Doubling the best zero-shot VLM

HOPLay-Docs-Bench · Overall Score

mean of the four track averages · 0–100

0255075100 best small open VLMGemini 3 ProQwen3-VL-8B ftDocLayout-VL-4BDocLayout-VL-8B Qwen3-VL-8B-it · 31.6 (open VLMs span 10.0–31.6) Gemini 3 Pro · 37.2 — best zero-shot Qwen3-VL-8B fine-tuned · 45.8 DocLayout-VL-4B-it · 71.8 DocLayout-VL-8B-it · 79.4 31.637.245.871.879.4
6.9 vs 77.5
hierarchical-track Open mIoU, Gemini 3 Pro vs ours — frontier VLMs can't build layout trees

Per track, 8B (Om · cI for promptable)

flat 86.7 · hierarchical 77.5 · curved 87.9 · promptable 70.2 (G-Eval 87.2)

Zero-shot closed-set (mAP@50:95)

PubLayNet 96.5 · M6Docs 76.9 · IndicDLP 77.5 — competitive with dedicated detectors trained per-dataset

DocLayout-VLMain results
DocLayout-VL · The deepest ablation

Architecture beats data scale

Overall score vs training-data fraction (Qwen3-VL-8B)

the gap never closes 25% data100% data VLM only (boxes) · 66.1 + SAM head · 76.8 + HGC · 78.1 full (+ struct losses) · 79.4 61.269.470.2 / 71.0
DocLayout-VLScaling
DocLayout-VL · The point

Layout as a queryable interface

DocLayout-VL teaser: a page becomes a controllable structured parse
One pass turns a page into a controllable structured parse: semantic nodes, hierarchy, pixel masks, and prompt-conditioned explanations.
A fund factsheet parsed into a layout tree
…and every node of that parse is groundable — here, a fund factsheet's full layout tree.
Takeaway #3 — with one [LAY] token per region, layout is no longer a preprocessing step — it's a model output you can prompt: tree, masks, explanations.
DocLayout-VLTakeaway
DocLayout-VL · SAM for documents
Segment Anything for Documents is here — DocLayout-VL
DocLayout-VLSAM for documents
DocLayout-VL · the team

Built by

VKVenkata Kesav Venna
Venkata Kesav Venna*
SBSrihari Bandarupalli
Srihari Bandarupalli*
ASAnirudh Srinivasan
Anirudh Srinivasan*
RRRaghuveer R
Raghuveer R
SMSai Madhusudan Gunda
Sai Madhusudan Gunda
RKRavi Kiran Sarvadevabhatla
Ravi Kiran Sarvadevabhatla
BharatGen · IIIT Hyderabad   * equal contribution

Thank you — questions?

DocLayout-VL project page QR
DocLayout-VL
doclayoutvl.github.io
BharatGenIIIT Hyderabad

DocLayout-VL (ECCV 2026)
BharatGen · IIIT Hyderabad