HTTP Archive and Web Almanac Data as Infrastructure Intelligence for Agent Builders

The HTTP Archive and Web Almanac describe the infrastructure conditions that browsing and extraction agents encounter on the live web. Their value extends beyond web-performance reporting. Read as infrastructure intelligence, their longitudinal data gives agent builders an empirical basis for retrieval budgets, parsing strategies, and fallback design, while runtime telemetry supplies the changing conditions that a crawl cannot capture.

What the HTTP Archive actually is and what it records

Steve Souders started the HTTP Archive in October 2010. It is part of the Internet Archive, a 501(c)(3) non-profit, and maintains a permanent, longitudinal repository of web infrastructure data. Its web performance testing infrastructure crawls millions of URLs each month, simulating browser visits and recording every resource loaded. The Chrome User Experience Report, or CrUX, supplies its URL lists, grounding the sample in real-world browsing.

The Archive has published the annual Web Almanac since 2019. Expert authors analyze its data across chapters covering performance, JavaScript, accessibility, security, SEO, and other dimensions of web construction. The 2025 edition analyzed 17.2 million websites. The underlying multi-petabyte dataset is queryable through Google BigQuery and includes raw HAR records, Lighthouse audit blobs, Blink feature usage tables, and custom metrics tables.

Why agent builders should read this as infrastructure data, not UX reporting

Page weight, JavaScript bundle size, DOM depth, and third-party chain depth describe the environment in which retrieval happens. They influence what an agent can fetch, when content becomes accessible, and how much execution precedes a parseable state. Accessibility structure gives agents more information to work with through semantic HTML, ARIA labels, and form names. SEO signals show how sites state their access preferences and explain their content to automated systems.

A pipeline that assumes immediately available HTML makes a different infrastructure bet from one that budgets for rendering and incomplete semantic markup. The Archive gives builders a population-level baseline to weigh those bets across target classes and see each failed extraction as part of a wider pattern.

A library handles a request, while the Archive describes the distribution of conditions across millions of targets. That distinction becomes important when setting timeouts, allocating rendering capacity, or designing fallbacks.

The page-weight and JavaScript conditions agents encounter on the live web

The 2025 performance chapter reports that Total Blocking Time rose even as INP improved. These measures describe different aspects of execution and responsiveness, so improvement in one does not imply that content becomes available sooner. For extraction systems, heavier JavaScript execution can make the wait longer before structured content is accessible.

Third-party dependencies make that wait harder to predict. The Archive records a median third-party inclusion chain depth of 3, as dependencies bring in further dependencies of their own. Consequently, the execution burden is not confined to scripts directly included by the page. Deferred and dynamically injected content also means that static HTTP retrieval can miss useful data even when the request shows no obvious sign of failure.

The design implication is to budget for payload variance. The Archive’s page-weight and JavaScript distributions provide evidence for rendering timeouts and decisions about when headless browser execution justifies its cost. Those decisions can differ across content categories. Responsiveness metrics remain useful here as signals about rendering predictability.

How CMS defaults shape the baseline configuration agents encounter at scale

The 2025 CMS chapter reports that CMS-driven sites account for over 54% of observed websites, up from a lower share in 2021. Their defaults therefore propagate across a substantial part of the retrieval environment. Payload, markup, and accessibility patterns are not simply unrelated choices made by individual sites. They often have shared structural origins in the platforms that generate pages.

The distribution also changes with the target population. Drupal has a higher share among top-traffic sites than it does overall, so a broad crawl and a pipeline focused on high-traffic sources encounter different platform mixes. Headless CMS architectures can separate content from presentation and potentially expose content through APIs. But they are still a minority in the Archive, and most CMS-generated content still comes inside full HTML stacks.

For agent builders, this supports parsing heuristics informed by the CMS distribution of a target domain class. E-commerce, news, and government sources need not be approached as structurally interchangeable. The chapter also documents pressure from AI-assisted workflows and LLM-based consumption to change platform defaults. Year-over-year tracking can distinguish a durable parsing assumption from one tied to a changing publishing stack.

Accessibility structure as the semantic signal layer for agent parsing

The 2025 accessibility chapter reports incremental progress in the median Lighthouse Accessibility score since 2019, but significant gaps remain. ARIA tooltip naming compliance improved substantially year-over-year, while other ARIA patterns remained poorly adopted. For agents, however, operation depends on the semantic structure available on the page.

Without alt text, an image lacks its textual description, while missing form labels make fields harder to identify reliably. Absent ARIA landmarks weaken the explicit boundaries between page regions. These accessibility gaps can cause parsing to fail even when the page makes sense to someone who can see it.

The Archive’s longitudinal data helps distinguish improving signals from persistent deficits that need lasting workarounds. That distinction is not guaranteed to move in one direction. The chapter notes that AI increasingly generates image descriptions and captions, but models trained on accessibility-poor code can also reproduce those patterns. Semantic quality therefore needs observation rather than an assumption that newer pages will be easier to parse.

Structured data coverage and the gap between schema presence and schema correctness

Structured data offers explicit machine-readable declarations beyond the cues supplied by HTML and accessibility markup. Its adoption is growing, but presence does not establish correctness. A sizable share of pages with structured data have validation errors that keep them from being eligible for rich results. For extraction pipelines, the corresponding risk is accepting malformed data simply because a JSON-LD block exists.

The 2025 SEO chapter describes llms.txt as “the web's latest experiment in speaking directly to AI systems,” allowing sites to signal how content should be ranked, read, and reused. The 2026 edition found that a meaningful minority of sites supplied the file, almost entirely as a 2026 phenomenon. Coverage remains incomplete across the wider web. The declaration is therefore a useful signal where present.

The practical response is to parse and validate structured data. Pipelines should check llms.txt while retaining HTML parsing fallbacks for sites without usable declarations. The Archive’s tracking of schema type adoption can also guide native vocabulary support. Actual prevalence offers a firmer basis for that work than a vocabulary’s prominence in specifications.

How the Archive now tracks AI crawlers as measurable infrastructure load

The 2025 Generative AI chapter broadened the Almanac’s coverage to include agents and their role in web infrastructure. It introduced AI fingerprints, including linguistic and stylistic markers in generated content and source code. It also documented emerging protocols such as WebMCP, designed to standardize how agents perceive and manipulate web interfaces. The subject is no longer only what agents consume.

The 2026 edition reported that GPTBot, ClaudeBot, PerplexityBot, and CCBot together represented a significantly larger share of crawler traffic than in 2025. It also reported AI engines as a notable fraction of pageviews on tracked sites in March 2026. These observations make automated activity a measurable component of infrastructure load. They create a feedback loop in which agents encounter a web increasingly shaped by automated consumption.

The SEO chapter provides a complementary access signal. It found that 85% of requests for robots.txt files returned valid responses, while describing robots.txt as a voluntary code of conduct. Fetch.ai’s Agentverse Almanac serves a different discovery function through a publicly readable, on-chain registry mapping agent addresses to endpoints, supported protocols, and capability metadata. The functions converge on reducing uncertainty for automated systems, although a registry describes declared destinations and the Archive measures the web’s observed conditions.

The security exposure agents inherit from the web's third-party script ecosystem

The 2024 security chapter found that the median percentage of scripts checked with Subresource Integrity remained very low on desktop and mobile. SRI lets browsers check that a script matches an expected hash before they run it. Meanwhile, at least 90% of pages load one or more third parties. With transitive dependencies and so few integrity checks in use, agents that run page scripts face a substantial supply-chain risk.

The Archive’s findings support sandboxing, output validation, and skeptical treatment of dynamically injected content rather than accepting rendered DOM state at face value. This is a structural risk baseline. The Archive documents conditions under which compromised dependencies can propagate, but it does not function as a threat feed.

Where the Archive's snapshot model reaches its limits for agents

The strongest objection is that structural data does not describe the full runtime environment. Bot-blocking, rate-limiting, and anti-agent countermeasures can change from week to week. A site may present an agent with access conditions that differ from those represented in a crawl. Annual analysis cannot resolve that variability, and monthly collection does not turn the Archive into live enforcement telemetry.

Declared policy is also different from observed compliance. A robots.txt file shows what a site asks crawlers to do. Neither the existence of a policy nor its parseability establishes the behavior of all actors encountering it.

Page weight, schema coverage, DOM patterns, and CMS distributions generally change on longer timescales than individual enforcement decisions. The Archive supplies evidence about those slower-moving conditions. Runtime telemetry must update that prior with the access and execution behavior encountered during operation, making the two sources complementary.

Tooling that operationalizes Archive conditions into agent pipelines

The tooling layer translates documented conditions into executable retrieval behavior. Heavy JavaScript, dynamic content, and unreliable structured declarations explain why a naive HTTP fetch is not a complete extraction architecture. Tool selection therefore needs to separate what rendering requires from how output is handled and how consistently the tools work across environments. A tool can address one part without resolving every failure mode.

Crawlee, built by Apify, provides a concrete example of operational consistency. It writes to a built-in dataset abstraction that exports to JSON or CSV and runs locally or on managed cloud infrastructure without code changes. That deployment continuity addresses a source of variability between development and production. Its dataset abstraction also gives retrieved results a defined output destination.

Those capabilities serve a narrower purpose than proving extraction correctness. Exportable data can still reflect incomplete retrieval or malformed source declarations. The Archive’s findings consequently remain relevant after a tool is selected, informing rendering budgets, validation, and fallback behavior. The engineering task is to connect documented environmental conditions to the pipeline mechanisms intended to handle them.

Using Archive data as a standing design input, not a one-time reference

Agent builders should use each Almanac release to revisit retrieval assumptions for the top five infrastructure dimensions most relevant to their target population. Page-weight findings inform timeout and rendering budgets, while schema adoption informs parser priorities. Accessibility findings guide DOM traversal and element identification, and SEO findings inform robots.txt handling and declared access signals. This makes the report part of a repeatable design process.

Between releases, BigQuery supports more specific questions than chapter-level aggregates can answer. Builders can query relevant domain sets, technology stacks, or content categories instead of assuming that the whole-web distribution matches their targets. Runtime telemetry then tests those assumptions against actual retrieval conditions. Persistent discrepancies identify where the population baseline needs a narrower interpretation or a live operational correction.

Year-over-year comparisons complete the method by showing which deficits are improving and which still require permanent workarounds. A fallback used because structure is widely missing may become less important as that structure improves. But even as the web improves overall, a target population that still has the same gap still needs those protections. The Archive compounds in value as successive observations reveal how the environment changes and which pipeline assumptions must change with it.