Vision-Language Models for Web Agent Navigation Tasks

Vision-language models for web navigation succeed or fail through three linked layers: perception, planning, and execution. Building reliable agents takes engineering across all three as well as a stronger base model. Page representation is now a tractable design choice, but long-horizon planning and dependable execution remain open problems that strong scores on structured benchmarks can obscure.

Why the perception-plan-act stack is the right frame for evaluating VLM web agents

A developer choosing how to build a web agent faces decisions about page inputs, action sequences, and reliable interaction. These decisions constrain one another at every step. A representation that makes an element easy to identify may still leave the planner without enough spatial context, while a sound plan can fail when the agent clicks the wrong target. Evaluating only the model leaves these dependencies unexplained.

VLM-based agents are now the main approach to navigation, replacing task-specific end-to-end models with large models that bring together language understanding, cross-modal alignment, and reasoning-based planning. Perception establishes what the agent knows about the page, planning turns that knowledge into an intended sequence, and execution succeeds when the sequence produces the intended result.

How agents perceive page state: the screenshot-versus-DOM tradeoff

Page representation determines input cost, spatial understanding, and the attainable accuracy of later decisions. No single format wins on all three. DOM and accessibility-tree inputs usually contain the most decision-relevant information, but their length makes them token-inefficient. Feeding large trees to a frontier model over and over during a multi-step task can cost a lot. Their informational strength therefore comes with a substantial context and cost burden.

Screenshots offer a different balance because they show the rendered interface, including visual hierarchies, affordances, and dynamic UI states. They can also reduce input-token cost compared with raw HTML dumps. Yet image inputs are not inherently compact. High-resolution GUI screenshots produce long token sequences in models that are not optimized for them, so the cost advantage depends on the model as well as the page. Hybrid representations aim to keep both structural and visual information, but they are harder to engineer and still involve a tradeoff between cost and accuracy.

Bounding-box overlays improved performance for visually capable models such as GPT-4o, while providing negligible benefit or reducing performance for others. Clutter can outweigh the value of the clues. The representation must therefore fit the model's ability to parse the information it exposes.

Textual grounding offers a useful counterpoint to visual annotation. Reframing selection as a multiple-choice question over structured text consistently yields the highest element-identification accuracy. That result has a narrow but important scope. Identifying an element is not the same as deciding when to use it or completing the task that requires it, so strong textual grounding does not settle the choice of representation for an entire agent.

How agents plan action sequences: the gap between instruction-following and web-dynamic reasoning

Even well-grounded perception does not guarantee a workable sequence of actions. Planning is where current VLMs most consistently break down on web tasks, and the deficit goes beyond a simple fine-tuning problem. Models can follow instructions and give correctly structured output while knowing too little about the environment. Those abilities must not be treated as interchangeable. The distinction becomes especially consequential when the agent must interpret changing state over a long sequence.

VLN-MME provides a diagnostic signal from embodied navigation. The evaluation framework found that models unexpectedly did worse when given step-by-step reasoning and self-reflection, even though they could still follow instructions. More explicit reasoning was not automatically better. The finding shows weaknesses in 3D spatial reasoning fidelity and context awareness, but leaves open whether these prompting techniques harm other web tasks. Its relevance is the separation it exposes between fluent reasoning output and reasoning grounded in the current environment.

SpaceVLN addresses that separation through Spatial Cognitive Memory and verifiable space–landmark stages. Its hierarchical approach lets the agent check progress and replan. Although this is an embodied-navigation approach, it illustrates an architectural response to the broader problem of maintaining state and checking progress across stages, without task-specific retraining.

NavGemini, published in January 2026, takes a complementary approach by decomposing navigation into subtasks managed through elaborate prompts and a prompt manager. The system operates zero-shot and outperforms prior methods despite using a less capable base model. This makes prompt architecture a substantive design variable. It also manages context length, addressing the accumulation of images and prior actions that raises cost as navigation history grows. Together, these approaches support architectural investment without establishing that the long-horizon planning problem is solved.

How benchmark scores reveal the planning gap without closing it

The strongest objection to a persistent planning gap is that some agents have reached or exceeded human-level performance on benchmark task sets. OSWorld illustrates how quickly the headline results changed. In April 2024, the best agent completed only a small fraction of tasks, while by December 2025, Simular's Agent S edged past the human baseline on the same set. That progress is substantial, but its interpretation depends on what the benchmark demands.

OSWorld 2.0 exposes the limits of that interpretation. Its long-horizon binary accuracy remains notably limited compared with the top OSWorld-Verified scores achieved by the same class of models. The contrast is the central evidence here. High performance on a structured task set and reliable completion of extended tasks measure different capabilities, so the former does not disprove weakness in the latter. Binary completion also measures whether the whole task succeeds, even when many of the steps along the way look reasonable.

WebVoyager is now considered saturated for top commercial agents, reflecting success on well-structured, short-horizon problems. WebForge and another benchmark emphasize challenges including automatically constructed environments, pop-ups, and ambiguous instructions. These benchmarks reflect a growing awareness that less structured environments bring out failures that familiar task sets hide. They can reveal the gap more clearly, but measuring a failure mode does not supply the planning or execution architecture needed to overcome it.

How execution breaks down: grounding errors, error compounding, and step-count inflation

Execution introduces structural failures even when the plan is correct. A page with a complex layout can have so many interface elements that it is hard to match the intended action to the right target. Agents can click empty areas or incorrect components. Such errors are not necessarily failures to understand the goal, because choosing the right action in principle and grounding it accurately on screen are separate capabilities.

An agent can also make a mistaken action without then correcting it. Without a feedback loop, errors compound through the remaining task, which makes isolated grounding accuracy a poor predictor of completion. Recovery must be part of the execution design. RegionFocus, presented at ICCV 2025, illustrates one narrower intervention by dynamically zooming into relevant regions. Narrowing the visual field cuts background clutter and improves grounding, but better target selection solves only part of the problem of recovering from failed actions.

Claude Sonnet 4.6 averages 45.8 steps per task on MyPCBench, the shorter-task comparison benchmark, while Claude Opus 4.7 averages 318.4 steps per task on OSWorld 2.0 long-horizon tasks. The comparison is therefore not a controlled measure of model efficiency, but it illustrates the extended execution burden that a more capable base model does not remove. Such tasks need architectures built for verification and recovery as well as longer sequences of actions.

Hybrid GUI-plus-code strategies respond by bypassing visual grounding for actions that can be scripted through code or API calls. Yet mode selection becomes another planning decision. The agent must determine when GUI interaction is appropriate and when direct execution is preferable, linking the third layer back to the second. Domain transfer brings another limit: models trained on web-centric screen-parsing data work reasonably well elsewhere on the web, but still fall short on native desktop and mobile UI toolkits. Reliability in a browser therefore does not establish reliability across interfaces.

Why adversarial pop-ups and prompt injection are an underweighted execution risk

Adversarial UI manipulation creates an execution risk that differs structurally from inaccurate grounding. The page itself is designed to mislead. An agent may perceive a pop-up correctly yet be diverted by its content, so improving element recognition alone does not address this failure mode.

The ACL 2025 pop-up attack paper found high attack success when adversarial pop-ups were integrated into OSWorld and VisualWebArena, together with substantial reductions in task success. These pop-ups targeted agents. Their effectiveness shows why standard task-completion benchmarks do not adequately test adversarial robustness. At the same time, environmental-injection research often assumes capabilities such as direct HTML manipulation, leaving a gap between controlled attacks and the range of attack vectors encountered on the open web.

Having a person watch all the time is impractical for agents meant to finish tasks on their own, so they lose the safety net that comes with human browsing. Current defenses do not solve this vulnerability. DUDE, the Deceptive UI Detector & Evaluator, combines hybrid-reward learning with asymmetric penalties and experience summarization in a two-stage defense, but remains a research prototype as of mid-2026. It is not deployment-ready, so its existence cannot be treated as evidence that the attack surface is controlled.

Browser automation tools can put a strain on web infrastructure by rendering pages over and over. As agents account for a larger share of web traffic, that resource burden is expected to grow more severe. This broadens the operational assessment beyond whether an individual agent completes its assigned task.

What the three-layer analysis tells practitioners about where to invest

For practitioners, the investment priorities follow from the uneven maturity of the three layers. Enough is known about perception choices today for engineers to make informed decisions. The appropriate screenshot, DOM, or hybrid approach depends on the model's visual capability, cost constraints, and whether the task needs spatial understanding or only element identification.

Planning needs work on architecture as well as larger models. Hierarchical decomposition, verifiable stage interfaces, and context management offer concrete ways to organize tasks that extend beyond a handful of steps. Strong structured-task scores do not remove the need to maintain state, check progress, and replan during extended work. Prompt architecture deserves attention because it shapes those processes.

Execution is the most underinvested layer, including its security dimension. Closed-loop feedback, dynamic zooming, and hybrid GUI-plus-code strategies show promise and are being worked on, but all are still at an early stage. Long-horizon task completion should therefore carry more weight in evaluation than single-step accuracy. For open-web production deployments, adversarial pop-ups remain a blocker to dependable unattended operation, and teams should assume the attack surface exists and build monitoring accordingly while research defenses mature.

The map is clearer than it was. The field's progress from low double-digit completion in 2024 to near-human-level structured-benchmark performance by late 2025 has left harder, more architectural work exposed in 2026. Long-horizon planning and adversarial robustness remain open problems. Practitioners who interpret benchmark headlines as deployment-readiness signals will mistake success under constrained evaluation for reliability under the conditions their systems must actually face.