Experimental Open Works

Research paper

The Binary Switch: How Websites Govern Autonomous Agent Access

Three measurements of published policy, enforced policy, and defended surfaces, and why operators withhold a vocabulary they already possess

Nik Nicholas, Experimental Open Works · 22 July 2026

Abstract

Autonomous agents now attempt tasks on the web that have consequences: booking, applying, paying, cancelling. We report three web measurements of how sites govern that traffic. First, across the top 500 domains by backlink rank (453 responded) and a hand-picked sample of 40 transactional sites (37 responded), not one site publishes any policy on any side-effecting action. The result is not a majority tendency but a census: 0 of 453 and 0 of 37, with all five side-effecting action classes unspecified on 100% of sites. Second, probing five identities per site, we find published policy and enforced policy are close to unrelated. On transactional sites where a browser is served, robots.txt permits an AI identity that the server then refuses in 19.3% of (site, identity) pairs, while robots.txt denials were enforced in 0.0% of pairs; 45% of transactional sites refuse an ordinary browser string outright, before any policy is consulted. Third, classifying 373 robots.txt files into a risk taxonomy shows that operators do possess a granular vocabulary and use it precisely: transactional sites fence inventory and cart state at 4.5 times the rate of the web at large, and account, pricing and API surfaces well above it. For AI crawlers the same operators discard that vocabulary: of 136 sites naming an AI crawler, 116 give a blanket Disallow: / and only 20 give targeted rules. We argue the barrier to graded agent access is therefore not an absent vocabulary but absent enforcement and absent attribution, and we give a theoretical account of why graded access is nonetheless achievable: the risks operators actually price are defeated by counting against an accountable principal rather than by inferring intent, so the correct unit of policy is the principal, not the request. We report two measurement defects found and corrected mid-study, one analysis error, and an undiagnosed residual sensitivity, and we bound our claims accordingly.

Headline result

Nothing on the web says what an agent may do. Where sites do say something, their servers largely ignore it. And where sites write careful, granular rules, which they do constantly, they stop doing so the moment the reader is an AI.

Contributions

1. A census of published agent policy over 490 responding domains, showing that coverage of side-effecting actions is zero, and that the specification fragmentation motivating much of the current standards work has essentially no deployed instances.
2. A paired measurement of declared against enforced policy over five identities per site, quantifying the disconnection between robots.txt and the layer that actually decides.
3. A risk-taxonomy classification of 373 robots.txt files showing that granular policy already exists, is aimed at the predicted risks, and is withheld from AI crawlers specifically.
4. A theoretical argument that the intent-unobservability objection to graded agent access is misframed, and that the correct unit of policy is the accountable principal, not the request.
5. A candid account of two instrument defects and one analysis error found during the study.

1. What sites publish

The surveyor issues seven requests per domain, to /.well-known/agent-policy.json, /agents.txt, /agent-manifest.txt, /ai.txt, /llms.txt, /robots.txt and /, then normalises whatever it finds into one model of seven action classes.

Adoption of agents.txt in either dialect is zero, so the incompatible-dialect collision that the specifications create, a genuine hazard on paper, occurs zero times in 490 responding domains. RSL, publicly backed by a large body of publishers, is deployed on one site. Our llms.txt figure of 9.3% sits close to an independently published figure of 8.7% for the top 1,000 as of June 2026, which is the best evidence available to us that the instrument measures what it claims to.

2. What sites enforce

Five identities, one GET each per site: an ordinary Chrome browser string as a control, GPTBot, ClaudeBot, PerplexityBot, and a plain bot that declares itself a bot without declaring itself an AI. On transactional sites, policy is frequently never reached at all.

The last row carries the interesting information. Declaring yourself an AI bot attracts refusal; declaring yourself an unremarkable bot mostly does not. Identity is already being used as a filter, and it is being used for exclusion only.

Read the third row twice. No transactional site in this sample with a robots.txt denial actually enforced it, while a fifth of the time the file explicitly welcomed a bot that the server then turned away. The published file and the component that decides are separate systems maintained by different people. The interstitials we received attribute to Akamai or a comparable WAF (33 responses), Cloudflare (13) and DataDome (4), so a substantial share of these decisions were not made by the site operator at all.

3. What sites protect

A site's Disallow list is its own statement of what it does not want automated traffic touching. Classifying those paths by risk category shows which risk each operator is actually pricing, without asking anyone anything. Classification is by path segment rather than substring, so /search matches and /researcher does not.

Transactional sites defend inventory state at four and a half times the general rate: carts, baskets, checkouts, bookings, reservations, seats and holds. They defend pricing, search and account surfaces substantially more heavily. They do not defend content or media any more than anyone else. This is not an inference about what operators fear. It is what they wrote down, in their own files, before anybody asked them.

The same operator who writes twelve carefully scoped Disallow rules for everyone else writes one line for GPTBot. This cannot be attributed to a missing vocabulary. The file format in which the switch is written is the same file format in which the dial is written, on the same server, for the same crawler, and the dial is two lines longer.

The unit of policy is the principal, not the request

The standard objection to graded agent access is that grading requires knowing why a request was made, and motive is not carried in the request. An agent booking a room, an agent extracting a pricing table, an agent walking a checkout flow in order to clone it, and an agent holding every middle seat on a flight so that only aisle and window seats remain purchasable all emit identical requests. We argue this framing is wrong. Every risk on that list is defeated by counting against an accountable principal, not by inferring a motive.

• Seat blocking works only because the two sessions holding the seats are anonymous and unlinked. Bind both to one verified end-user and "six held seats, no purchase" is arithmetic.
• Price extraction yields to volume and conversion ratio per principal: forty lookups and a booking is not ten thousand lookups and nothing.
• Inventory holds in general: "one active hold per principal" is expressible, countable and enforceable, and entirely meaningless without a principal to count against.
• Flow cloning is not defensible and arguably not worth defending. Anyone can screen record a checkout.
• Training on content is orthogonal. That is the licensing layer, and RSL already has a vocabulary for it.

So the predicate changes. The wrong predicate is may you book, which requires knowing why the agent is here. The right one is you may hold one seat, as this verified principal, at this rate, with this consequence if you abuse it, which requires only identity and arithmetic. Arithmetic has always been available. Cryptographic agent identity and scoped delegation, which supply the principal, became available in 2026. That is why the binary switch was, until now, the correct engineering decision rather than an oversight.

The third experiment supplies the support this argument would otherwise lack. If operators feared an unbounded and unarticulable range of agent behaviours, the categories they fence would be diffuse. They are not. Transactional operators fence inventory state at 4.5 times the general rate and content at 0.9 times, which is to say they price precisely the risks that per-principal counting defeats, and do not price the one risk that counting cannot address.

What this means

The missing piece is not a vocabulary. It is enforcement plus accountability. A new file format, however well designed, changes nothing on its own, because nothing reads a file at the moment of decision: the decision is made in a WAF, frequently one operated by a third party, and the WAF has never read the file.

Your robots.txt says GPTBot is welcome. Your WAF refuses it. You are turning away the assistant your customer sent to book a room, and nobody at your company knows.

Limitations

The paper devotes a full section to these, because several were found by us during the study and would not otherwise be visible. In brief:

• Two measurement defects, found and corrected mid-study. A fresh HTTP client per domain removed connection pooling across the seven probes to each host, understating robots.txt detection at 31% before the fix and 76.2% after. And RSL discovery matched any HTML rel="license", so Wikipedia's ordinary Creative Commons link was misread as an RSL declaration, producing seven false findings; requiring type="application/rsl+xml" made the finding vanish entirely. A measurement instrument that manufactures findings is worse than none.
• An analysis error, corrected. The first declared-against-enforced figure was 33%, which wrongly counted sites that refuse all traffic from our address as sites contradicting their own policy. Restricting to sites where a browser is served gives 19.3%.
• Residual concurrency sensitivity, undiagnosed. On one fixed 60-domain slice, 40, 46 and 51 sites answered at concurrency 20, 10 and 6. All reported surveys were run at 6.
• A single vantage point, one IP address and one geography. Blocking is often address-reputation dependent.
• Homepages only. Deeper transactional paths are likely more heavily defended, so the practical picture is probably worse than measured.
• Our user-agent strings retain the real product token but append our own identifier and a contact URL, so a rule matching a full user-agent string exactly will not fire. Blocking rates here are a floor, not a ceiling.
• Majestic backlink ranking over-weights infrastructure domains that would never publish an agent policy. The transactional sample exists to correct for this, and returns the same answer.
• Small transactional samples: 40 sites, 28 with robots.txt. The 4.5x inventory result is convincing as a direction, not as a precise figure.
• other is the largest taxonomy bucket, covering 77.7% of sites, so no claim of exhaustive classification is made.

Ethics

One GET per identity per site, no retries, no attempt to bypass any access control, public homepages and well-known policy paths only. Requests to the same host were separated by 400 ms, with never more than one in flight per host. No authentication was attempted and no state was created or modified on any site. Every user-agent string carried a contact URL, so any operator reading their logs could identify the source and reach us.

Cite

@techreport{nicholas2026binaryswitch,
  title       = {The Binary Switch: How Websites Govern Autonomous Agent Access},
  author      = {Nicholas, Nik},
  institution = {Experimental Open Works},
  year        = {2026},
  type        = {Research paper},
  url         = {https://xowx.org/research/}
}

The full method, all tables, the risk taxonomy keyword lists and the reproducibility notes are in the LaTeX source. Raw JSON records, the domain lists and the analysis are available for research on request through the contact page.