Agent Action Space Design for Browser-Based Task Completion

Browser-based task completion depends on the actions an agent can express and the page evidence it can use to ground them. Action space design is therefore a tradeoff between predictability and flexibility. Fixed primitives make behavior tractable and comparable, while code generation reaches interactions that predefined schemas miss. Reliable designs match that freedom to the task, the perception layer, and the acceptable security surface.

Why the action space is the most consequential design decision a browser agent team makes

The action layer is the set of tools a model can call, including click, type, scroll, navigate, wait, extract text, take a screenshot, run JavaScript, and upload a file. Playwright or Puppeteer drives the browser underneath that interface. Every interaction passes through this boundary. Its design determines both what the agent can do and how it reasons about the next step. In legacy portals, EHRs, government systems, freight TMS, and supplier portals without APIs, those available actions must span interfaces built for humans. An interaction missing from the action layer is not simply harder to perform. It may be impossible by design.

What a fixed action space actually looks like and why it became the default

WebChoreArena uses 15 actions in BrowserGym, which cover basic actions, tab operations, page operations, and messaging. The engineer defines the observable state and available actions, the model selects an action, and the runtime executes it before returning the next state. BrowserGym formalizes this loop as a Partially Observable Markov Decision Process, with each step returning a structured observation tuple. The model chooses within an established vocabulary.

OS-Symphony identifies two governing principles for this design, which are cross-platform abstraction and conciseness and generality. These principles keep the vocabulary pragmatic and broadly applicable without accumulating redundant actions that add context overhead and cognitive load. Actions that are too specialized also limit what the agent can do. A smaller contract is easier to inspect. Fixed sets became the default because they support auditability, bounded execution, and reproducible evaluation, provided engineers anticipate the interactions tasks require. WebChoreArena, WebArena, and Go-Browse use well-defined, stable action sets that make comparisons across agents and models tractable.

Where fixed action spaces break: the edge-case problem that code generation solves

Browser Use encountered the limits of predefined actions in production when drawing a signature and looping through comments each required new actions. These were not merely different values for an existing click or type call. They required interactions beyond the available vocabulary. Under a fixed design, an engineer must add the missing operation or accept failure on the task. Code generation changes that dependency by allowing the model to compose an interaction at execution time.

A cookie button can be visible on screen yet absent from the state sent to the model because the accessibility tree omits it or processing drops it. The button exists, but the agent cannot ground its click. An EHR dropdown inside a cross-origin iframe, itself nested inside a closed shadow root, makes the structural problem more explicit. Fixed-state heuristics cannot anticipate every website’s DOM topology. Browser Use moved from Playwright to direct Chrome DevTools Protocol access to inspect closed shadow roots, illustrating why a broader action interface sometimes needs deeper browser access as well as generated code.

How code generation expands the action space and what that expansion costs

Browser Use introduced JavaScript execution in September 2025 and experimented with persistent notebooks in October 2025. Its Hermes agent later replaced 12 browser tools with a single browser_exec action, with both versions solving the same task suite. The apparent interface shrank while its expressive range grew. Variables persist across steps, so the model can check results along the way and keep working through code. Direct CDP access extends that control beneath Playwright’s abstraction layer, where the model can choose a screenshot for one question and a DOM query for the next.

Generated code produces longer outputs than primitive selection, so token use accumulates into latency and expense across long-horizon tasks. Arbitrary execution also exposes users to vulnerabilities from untrusted instructions. Expressiveness does not require unrestricted execution, however. WebChoreArena uses BrowserGym’s action-mapping function to constrain generated code to its 15-action set, providing a middle position between code syntax and arbitrary browser control. Action mapping and sandbox constraints limit what generated programs can run and which interactions they can use.

How the perception layer — what the agent sees before it acts — reshapes the action space tradeoff

An agent can only use its action vocabulary effectively when its observation provides a reliable target. A DOM snapshot filtered to interactable elements is compact, but it can omit dynamically rendered or shadow-DOM content. An accessibility tree supplies semantic labels, roles, focus state, and validation messages, using less transmission overhead than pixels while remaining incomplete on hostile pages. A viewport screenshot instead exposes what is rendered, regardless of DOM structure, at a higher token and per-step latency cost. These are different grounding contracts.

The environment determines which contract holds up. Government forms, insurance portals, procurement systems, and bank interfaces can contain framesets, table layouts pretending to be forms, and embedded PDF-ish elements. On such pages, screenshots degrade more gracefully than DOM serialization, making vision-first grounding more than an occasional fallback. Accessibility-tree extraction also adds latency across repeated steps, and completeness varies across platforms. Browser Use’s April 2026 Browser Harness direction lets the model control what it observes through CDP, using screenshots for spatial reasoning and DOM queries to find elements. This shifts perception from a fixed preprocessing choice into something the agent can adapt during execution.

What benchmark results reveal about where each design fails in practice

Go-Browse, published at ICLR 2026, shows that structured exploration data improves a fixed-primitive agent enough to outperform GPT-4o mini on WebArena, while leaving a substantial gap on the hardest tasks. The important distinction is the task distribution. Results on well-scoped, single-site interactions do not establish reliability on long-horizon, cross-site, structurally complex workflows. Nor does that gap alone prove that arbitrary code would solve them.

WebDS, published at ICLR 2026, makes this transfer problem explicit. Agents with strong WebVoyager results perform much worse on its multi-step, multi-site, tool-using data science workflows, which expose poor information grounding, repetitive behavior, and shortcut-taking. WebChoreArena, published at COLM 2026, isolates labor-intensive, multi-record procedures across Shopping, Shopping Admin, Reddit, GitLab, and multi-site domains and finds substantial room for improvement even among powerful models. Simpler benchmarks can conceal these procedural weaknesses. The implication for production evaluation is that a high score on a bounded task set cannot establish suitability for a regulated-industry portal workflow with different structural and sequencing demands.

Security consequences that follow directly from how broadly the action space is defined

The same freedom that helps an agent handle unfamiliar interactions enlarges the surface available to an adversary. An action-capable agent can turn untrusted instructions into external operations, unlike a standalone language model without those tools. The action contract thus helps set the blast radius of a successful injection. Code generation makes that contract much broader when execution is arbitrary. Security is consequently part of action space design itself, rather than a separate review conducted after capabilities have been chosen.

The design response is to apply least privilege before expanding capability. Each agent should receive the narrowest action space that covers its task, so a read-only research agent should not have write or submit actions. Credentials belong in application code, outside the agent context. When one agent combines untrusted input, sensitive data, and external actions, per-action human approval is required, while every output should be validated in trusted code outside the agent’s execution context. BrowserGym’s sandboxed action-mapping approach shows how a framework can limit execution to an allowed subset. The boundary can be enforced by controlling which operations are allowed as well as checking the generated text.

How MCP and skill libraries change where the action space boundary sits

Playwright MCP separates action delivery from the agent runtime by exposing structured browser automation through Playwright’s accessibility tree without a vision model. It supports Chromium, Firefox, and WebKit, with optional capabilities behind flags for network mocking, storage, tracing, video, PDF generation, and test assertions. The runtime need not own the tool implementation. This makes action spaces composable and swappable across task types without changing the core reasoning loop. It does not remove the need to decide which operations are permitted or whether the tree adequately represents the page.

Skill libraries move the boundary upward, from individual operations to reusable, parameterized action subsequences for known workflows. PolySkill keeps an abstract goal separate from the way it is carried out on a given site, so it can be reused on seen and unseen websites within the same domain. SkillWeaver discovers reusable website procedures and distills them into callable APIs. WebXSkill, published by Microsoft Research in April 2026, addresses the gap between textual guidance and executable but opaque code through grounded execution and guided, step-by-step modes. Reuse introduces another cost to weigh, however. Whether retrieval overhead and token cost pay off at inference time on long-horizon, cross-site workflows remains unsettled.

A framework for choosing between fixed primitives, code generation, and hybrid designs

The choice should start with the task distribution and page environment, how much latency and cost the budget allows, and how broad a security surface is acceptable. Fixed primitives are an appropriate starting point when tasks are well-scoped and interactions are knowable in advance, including form filling, navigation, and structured extraction on known portals. They also work well for workflows whose action records must be easy to explain and environments that rule out arbitrary code execution. WebChoreArena’s 15-action BrowserGym configuration offers a concrete starting point for that profile. Predictability is useful when the workflow itself is bounded.

Code generation earns its overhead when the interaction set cannot be enumerated beforehand, as with signatures, variable-length comment threads, and conditional behavior driven by page content. Structurally hostile pages strengthen that case, especially when custom widgets, closed shadow roots, or cross-origin iframes prevent ordinary targeting. The runtime remains part of the decision. A proven coding harness with tested compaction and error handling lowers the risk of loop bugs that comes with custom implementations. The same task analysis should guide perception, with accessibility trees for well-structured pages, screenshots where the DOM representation is misleading, and CDP-level access when the model needs to choose how to observe the page.

A hybrid design can combine these layers without granting every capability to every task. Its security posture should be set first, with access expanded only as needed and human-review pauses at high-consequence action boundaries. Asteroid’s full-stack platform combines perception, action, orchestration, and human-review pausing in one runtime, offering an option for teams that prefer configuring and monitoring those layers to assembling them independently. Skill-library retrieval still needs to be worth its inference cost, simpler benchmarks leave open questions about long-horizon reliability, and code generation still has a broader security surface wherever arbitrary execution is allowed.