DOM Parsing Strategies for Reliable Agent Data Extraction

Reliable agent data extraction means moving past the assumption that raw HTML is a workable input for language models. The right posture is a layered one: compress before reasoning, check for structured data before parsing, and match the format to the downstream consumer. Each technique has a place where it succeeds and a place where it fails.

Raw HTML was designed for browser rendering, not for LLM reasoning. Feeding it directly to a model imposes a compounding set of costs that make reliable extraction impractical at scale. Research on index-based web content extraction establishes the signal-to-noise problem in concrete terms: even after stripping the vast majority of JavaScript and CSS bytes, the remaining HTML still contains on the order of thousands of tokens per page. Those token counts continue trending upward. Low content density is structural, not incidental. Redundant scripts, tracking tags, navigation chrome, and cookie banners are baked into the document before any content appears.

The numbers make the constraint vivid. In the SWDE benchmark studied by Co-Scraper, the longest raw HTML reaches 219,000 tokens. That input exceeds practical context-window budgets outright, and it degrades model reasoning even when it fits. The degradation is not hypothetical. The model attends less reliably to exactly the region where the target data often sits. Noise compounds the problem rather than simply adding to it.

The failure mode is not a hard crash. The problem is not that raw HTML is occasionally too large. It is that raw HTML is the wrong default, and the costs of that default are paid quietly, across every page, at every stage of reasoning.

How query-aware DOM pruning changes the cost structure of extraction

Compressing the DOM before any LLM reasoning, rather than asking the model to reason through the noise, is the central architectural insight shared by the leading extraction systems of 2025 and 2026. Several prominent extraction systems converge on the same compress-first principle. The shift is architectural because it moves the cost of filtering from the model, which pays per token, to a preprocessing step that runs once before any reasoning begins.

Co-Scraper, from the Shanghai Artificial Intelligence Laboratory and USTC, demonstrates the principle in a fully specified two-stage framework. The first stage converts the raw DOM into a simplified, flatter representation and identifies query-relevant nodes. The second stage prunes redundant branches before the model ever sees the document. The pruning is query-aware, not generic. A navigation menu is irrelevant when extracting product prices and essential when extracting site structure.

What a flatter hierarchy means in practice is worth specifying. Co-Scraper's pruning reduces deeply nested structures to addressable, manageable segments that preserve essential document structure while eliminating branches that carry no information for the target fields. The model receives a smaller document, but not a shredded one. The hierarchy that remains is the hierarchy that answers the query.

Pruning carries a risk that deserves acknowledgment. The strategy induction step confirms stability across instances rather than committing to one page's structure. The result is a scraper that extracts from similar pages, not a one-off parse, with an F1 score of 94.78% on SWDE and a reuse success rate of 90.39% by the paper's report.

What Prune4Web adds: using an LLM to write the pruning logic rather than run it

Co-Scraper uses an LLM to score and filter DOM elements. Prune4Web shifts the LLM's role from DOM reader to script author, and that shift separates the expensive reasoning step from the cheap repeated execution step. The LLM generates key parameters, semantic keyword-weight dictionaries, for executable Python scoring scripts that dynamically filter DOM elements based on semantic cues from decomposed sub-tasks. The LLM reasons once to produce the filter. The filter then runs programmatically at low cost across many pages. This makes large-scale grounding practical in a way that asking the model to examine every candidate element on every page cannot match.

The effect on grounding is elimination of attention dilution. The model is never asked to ground an action against tens of thousands of candidate elements, because the programmatic filter has already reduced the candidate set by a large factor before grounding occurs. Presented at AAAI 2026, Prune4Web's approach raised grounding accuracy to 88.28% on a low-level task grounding benchmark. The magnitude of the gain illustrates how severely attention dilution degrades performance on full DOMs.

The practical implication is a clean division of labor. For agents that must ground actions on live pages, such as click targets, form fields, and interactive elements, Prune4Web's grounding-focused design is the more relevant pattern. For bulk extraction with reusable wrappers, Co-Scraper's scraper-synthesis approach fits better. The choice turns on whether the agent is acting on a live page or harvesting data from many similar pages.

Index-based selection as an alternative when pruning is not enough

Pruning reduces noise. The model outputs index pointers, not generated text. The task becomes discriminative rather than generative, and that change decouples latency from document length.

The speed advantage follows from the mechanics of generation. The structural awareness is also preserved in a way that distinguishes this method from chunk-and-rerank approaches. Index-based segmentation respects HTML structure when partitioning, so the segments are addressable units that carry their position in the document hierarchy.

The trade-off is localization. Index-based selection works best when the relevant content can be localized to discrete segments. In a pipeline, index selection fits naturally as a post-retrieval processing step before an LLM reads the content. It combines with upstream pruning rather than replacing it: prune first for noise, then index for speed.

When to prefer structured data already in the page over parsing the DOM at all

Before committing to any pruning or extraction strategy, an agent should check whether the page already exposes its target data in structured form. Many pages do. Embedded JSON blocks and background API responses are more stable, cheaper to parse, and structurally cleaner than anything the DOM's visible markup contains. The check is quick, and when it succeeds it removes the entire DOM parsing problem.

The most accessible signal is the script type="application/ld+json" block. These blocks embed Schema.org-structured data intended for crawlers: prices, products, events, reviews. They require no selector logic to extract, and they survive layout refactors that break CSS-based parsers. Background XHR and fetch calls are a second source. They often return the exact JSON that populates the visible DOM, and inspecting network traffic during page load can expose an endpoint the agent can call directly, bypassing the rendering and parsing stack entirely.

The stability argument matters as much as the convenience. The path is not always available, of course. The practical framing is to check for embedded JSON and documented background endpoints first, and escalate to DOM pruning when those signals are absent or incomplete. The layered framework means no single technique is universal, and the cheapest techniques belong first in the escalation order.

Format choice after extraction: what HTML, Markdown, and JSON each preserve and what they lose

The public argument over whether agents should consume HTML, Markdown, or JSON presents the question as a contest with a single winner. It dissolves when the format decision is matched to the downstream consumer rather than chosen once for the whole system. Each format preserves something the others lose, and the right answer depends on what the next stage needs to do.

Markdown reduces token consumption substantially compared to raw HTML. Benchmark evidence also shows that table extraction and RAG QA accuracy can improve when models ingest Markdown over raw HTML, with RAG pipelines seeing up to a 35% accuracy improvement in one analysis of ingestion formats. For word-level reasoning, Markdown is the efficient, effective choice.

HTML preserves structural semantics that Markdown flattens: nesting, heading hierarchy, table relationships. JSON is a different thing entirely. It is typed, machine-parseable, and unambiguous about field identity. It is not a reading format for an LLM reasoning about a page. It is a handoff format between pipeline stages, and it belongs at the boundaries where one component passes structured output to another.

The practical rule follows. Serve clean semantic HTML, with valid Schema markup, to the extraction layer. Use JSON at handoff points between agents or pipeline stages. The format is a function of the consumer, not a preference to settle once.

How DOM structure changes silently — and why selector durability requires semantic anchors

The most dangerous DOM changes in 2026 are not broken selectors but semantic drift. A price field still exists at the same CSS path, but its value now reflects a regional price, a sale state, or a user-segment-specific figure depending on configuration flags or client-side logic that never touches the HTML structure. Sites now run experiments through CMS rules and feature flags rather than HTML rewrites. The selector keeps matching, the data is wrong, and the agent reports it with high confidence.

Semantic drift is distinct from traditional DOM breakage, where a node is gone or moved and the selector fails loudly. Semantic anchors, such as data-* attributes, aria-label, ARIA roles, and Schema.org markup, are authored to communicate meaning, not presentation. They survive layout refactors that break class-name selectors, because a redesign changes visual structure, not semantic intent.

The version-aware selector pattern demonstrates the durability lever. GroupBWT's case study on a B2B e-commerce site showed that implementing version-aware selectors reduced downtime by a large margin over two months, even as the frontend deployed more than a dozen hotfixes. Prices that appear only after scrolling and fields that populate after user interaction require the agent's fetch layer to trigger rendering before the extraction layer runs. That is not a smarter selector. It is a prerequisite for any selector to have anything to match.

Security risks that DOM parsing introduces into agent pipelines

When an agent parses and acts on web content, the DOM is not just a data source. It is a potential instruction channel. Failing to sanitize it at the extraction layer exposes the entire agent to hijacking, because the content the agent reads can carry instructions the model will treat as legitimate.

A May 2025 security evaluation of browsing agents demonstrates the attack concretely. Through white-box analysis of the open-source Browser Use project, the researchers showed that untrusted web content can hijack agent behavior via prompt injection embedded in DOM content. A disclosed CVE and working proof-of-concept are documented in the paper. A scraped page that contains malicious instructions in visible text, hidden elements, or meta tags can redirect agent actions without tripping any obvious alarm.

The defense-in-depth response maps directly to the extraction layer. Prompt injection via the DOM is especially hard to detect because it looks like content, not code. Extraction strategy and security posture are coupled, and practitioners who treat them as separate concerns create a gap attackers exploit.

Composing the techniques into a layered extraction pipeline

The techniques are not interchangeable alternatives to be evaluated once and chosen. They are complementary layers that each address a different bottleneck in the extraction pipeline and belong at different stages. The order matters because each stage only engages when the stage above it cannot satisfy the query.

Stage one is fetch and sanitize. Resolve the page to its rendered state, strip instruction-like content as a security measure, and check for embedded structured data such as JSON-LD and background API responses before engaging any DOM parsing logic. Stage two is query-aware pruning. If structured data is absent or incomplete, apply DOM pruning scoped to the sub-task semantics: Co-Scraper's flatten-then-prune pattern for bulk extraction with reusable wrappers, or Prune4Web's LLM-authored scoring scripts for action grounding on live pages.

Stage three is index selection. After pruning, apply index-based segment selection to identify the relevant content positions before passing anything to the LLM reading step. Stage four is format and handoff. Convert extracted content to the format appropriate for the downstream consumer: Markdown for LLM reasoning, clean semantic HTML for structure-sensitive extraction, JSON for agent-to-agent handoff. Stage four is always required. The others are conditional.

Tools like Firecrawl operationalize several of these stages as managed endpoints, offering JavaScript rendering, clean Markdown or structured JSON output, and schema-defined extraction. The same principle applies to extraction APIs that abstract the fetch and format conversion layers. What no automated stage catches is semantic drift. Monitoring the meaning of extracted fields, not just their presence, requires a validation step that compares values against known-good baselines or cross-checks them against independent signals. The pipeline handles structure and noise. Judgment about whether the data means what it should remains a human responsibility outside it.