Marketfauna · Agent access

What the sites you fetch actually do to your bot

For operators of small, honestly identified crawlers and agents. A robots.txt tracker tells you what a site says. An access reading tells you what the site returned to an identified request today, site by site, with the signed and unsigned cases kept apart and the cause left open where it is not established. The setup service puts a signed identity on your collector to the current Cloudflare profile, with the tests to show it.

For operators

Agent Access Reading

$149 per panel, up to 25 sites

You name the sites your collector needs, one public URL per site (the root by default), and the User-Agent it sends. Within 48 hours of agreed scope you get a dated reading, readable and as JSON, with response metadata, headers, hashes and diagnostic evidence retained.

What is probed, exactly: requests sent by us, from our network, carrying by default your User-Agent string followed by our own reading token, which names us and the name you ask us to put on the reading, so the site sees both whose agent this is and who is asking. We send your string on its own only with your written authorisation, and our own bot's string if you give us none. For the signed condition, our own signing key and published directory; recognition by the destination is not assumed. Three attempts per condition, plus one control request per site from a generic, honestly labelled HTTP client (never a browser disguise), which helps tell a refusal of the User-Agent we sent from a refusal of any non-browser client at our address; one request is a clue, not a measurement. A User-Agent is not your egress, session or signing identity, so this is a reading of how the sites treat an identified request shaped like yours, not a reproduction of your bot. The instrument is published and needs only Python 3.7 or later, with no other packages; run from your side it makes the unsigned and control requests, since the signing key stays with us. If you run the same scope from the network your agent uses, we set the two readings side by side at no extra charge; refusals by address range are common and cannot be seen from our side alone.

Declared policy per site: robots.txt for your agent name, for our own tokens, for unnamed agents, and for GPTBot and ClaudeBot as a comparison, from the file as served that day
Observed response to an honest, identified request: status, challenge headers and the reported server, three attempts per condition per site and at most ten requests to any host, robots.txt honoured throughout and evaluated under RFC 9309 for the exact URL, crawl delays respected between attempts up to 30 seconds (a longer requested delay is waited as 30 seconds and the row says so), every redirect of a page request checked against robots.txt before it is followed; if robots.txt is refused to us (401, 403, 429, a challenge header or a challenge page) or cannot be reached, the page is not requested and the row says so, while a 404 or other 4xx is read as no restrictions under RFC 9309 with the basis stated; a 200 is reported as an HTTP observation unless a content marker confirms the page arrived
Signed versus unsigned, measured separately: whether presenting a Web Bot Auth signature changed the observed status at each site, reported as observed and never inferred
The next step per site where you were refused: who has to act, what they need in hand, and how to tell it worked. A documented API or a vendor's bot programme where one applies, the site owner where none does, or exclusion. We never suggest disguising the agent, rotating addresses or solving challenges
Where a free tracker or a few-cent automated checker already answers the question, we say so and link it instead of charging you for it. What those cannot give you is your own agent's identity, per-attempt evidence, sourced vendor clues in stated tiers, and a specific route per refusal

Pay $149 by card Agree scope first

A reading is an observation from our network on one day, not a guarantee of what your network sees tomorrow. HTTP 200 is not permission: whether a site's terms allow your use is a reading of the terms, and the packet says where that reading is still needed.

For operators

Web Bot Auth Setup

$249 pilot price, fixed scope

A signed identity for one collector, to the current Cloudflare profile: RFC 9421 Ed25519 request signatures, a signed key directory on infrastructure that can set response headers (a free Worker on your own account), and the directory application prepared for you to submit.

Scope of the pilot: one Python collector, one directory deployment on your own Cloudflare account, integration done in your environment with the repository and account access we agree up front, and a handoff with the test evidence. You generate and keep the private key. Other languages are quoted separately. Retesting after Cloudflare's decision is included once; ongoing support is not.

Signing integrated into your collector (Python reference implementation, MIT) with expired and tampered cases failing in tests
Key directory deployed and verified live: correct media type, thumbprint key id, signed response, no-store caching
Signing controls tested against our separately implemented verifier; Cloudflare diagnostic results included. Before registration, Cloudflare can reject the request as an unknown key, which does not establish signature acceptance
Application text and field values prepared; you submit from your own account

Pay $249 by card Ask about setup first

This does not obtain verified-bot status. Cloudflare decides that in its own review, registration is necessary but not sufficient, and no site is obliged to admit a verified bot. Our own collector's application has been waiting for review since 15 September 2026; we will say when that changes.

What a reading looks like

In our 15 September 2026 reading of 24 sites, our parser found no root disallow for our unnamed identified bot at 20. Four of those sites returned HTTP 403 on every signed and unsigned attempt; a fifth returned 403 intermittently. A sixth produced transport failures, whose cause we did not identify. Signed and unsigned status sequences matched in this run. We did not establish that any site recognized our signing key. These are observations from our network, not a test of a customer's collector.

One row from that reading, as delivered under the method of that day. When we read this site again on 18 September 2026, robots.txt itself was refused (403 with a challenge header), and the instrument of that afternoon still requested the page, which was wrong. Since version 3, later the same day, a site that refuses us at robots.txt is not asked for the page, and the refusal at robots.txt is the recorded observation.

Site	Declared policy	Observed, unsigned (3)	Observed, signed (3)	Next step
www.npmjs.com	robots.txt: no root disallow for our agent, for unnamed agents, or for GPTBot and ClaudeBot	403, 403, 403 with `cf-mitigated: challenge`, `server: cloudflare`	403, 403, 403, same headers	The npm registry publishes a documented API; the challenge is on the website, and the reading says which one your collector needs

Column	What it contains
Declared policy	robots.txt verdicts for the exact URL, for your agent, for our tokens, for unnamed agents, and for GPTBot and ClaudeBot; the file's fetch status and hash
Observed response	Status per attempt, challenge headers such as `cf-mitigated`, the reported server header as a clue and never as proof of cause, whether the final URL differed from the requested URL, and transport failures kept as their own class
Signed observation	The same requests carrying a Web Bot Auth signature, reported beside the unsigned ones, with what can and cannot be concluded from one key on one day
Next step	Per refused site: who acts, what they need and the test that shows it worked; a documented route where we found one, otherwise the site owner
Substitutes	Where BotWitness, StackScan or the site's own documentation already answers the question, stated plainly

Rules we hold ourselves to

robots.txt governs: if it disallows your agent or ours, or is refused to us, or cannot be reached, we do not request the page and the row says so. Honest User-Agents only: the identified request and one labelled control, no evasion, no challenge solving, no proxies. A refusal is reported with the evidence that produced it, and no server header is presented as proof of cause. A null result is reported as null. We do not scrape sites whose terms we have read as barring automated access.