Stop Blocking AI Agents: Manage AI Crawlers Without Blanket Rules

Somebody on your team wants to block AI crawlers. It's an understandable instinct.

Some of that traffic is a training scrape you never agreed to. Some of it is a prospective customer asking ChatGPT, Claude, or Grok whether your product can do the job, and the assistant fetching your pricing page to find out. Block all of it and you've kept a scraper off your content... and your product out of the answer.

The block doesn't even work very well. The polite crawlers announce themselves and respect robots.txt, so they were never the problem. The rest don't announce themselves at all. One independent test found Grok's live traffic arriving as ordinary Chrome from datacenter proxies, never under the crawler names listed for it. A user-agent rule can't find the assistants, let alone the scripts that borrow their names.

A user agent tells you what a client claims to be. It doesn't tell you what the client is about to do. The policy that works lets fetchers fetch and saves its judgment for whatever signs up. That's the job of a judgment layer, whether it's Dregs or another one.

Why Blanket Rules Fail

You are right to want the registration farm gone. Scripts still invent names, rotate inboxes, and open accounts overnight. That is signup bot abuse, and it belongs off the form. A one-line rule that blocks every automated client does not stop that farm. It blocks the traffic you meant to keep.

The rule that stops "AI bots" tends to stop everything else that isn't a person: the Pingdom check on /health, the preview bot that builds your card when someone shares a link on LinkedIn, the research crawler reading your public API docs, and the assistant a customer sent to read your pricing page. Those are false positives, and they land on your customers and your search presence, not on the pest you meant to stop. A puzzle on the same door makes the same mistake: it taxes the assistant, and a solver farm still walks through. CAPTCHA alternatives covers what to use instead.

Meanwhile the more ambitious of these pests have noticed the panic. Copying an assistant's user agent is cheaper than a residential proxy, and it looks like a helpful client in the logs. A session that calls itself an assistant and then burns a trial is not an assistant. It is a signup bot using a borrowed name. Knowing that a request is automated is the easy part. Deciding whether it is a problem is the part that matters.

How to Identify AI Crawlers

"AI crawler" is not one act. The large operators split their traffic by job and publish a user-agent token for each, which is the most useful thing to know when you are reading logs.

Then there is the traffic that borrows any of those names: signup bots, credential stuffing, scrapers hoarding content or inventory, and bot farms manufacturing fake accounts. Stop these, preferably before they become "users" in your table, whatever header they are wearing. The wider taxonomy, including the good bots that have nothing to do with AI, is in good bots vs bad bots.

Treat the names as a starting point, not a credential. Copying a user agent costs nothing, and not every useful assistant announces itself. Crawler directories list Grok user agents such as GrokBot, but an independent test found Grok's live traffic arriving with generic Chrome, Safari, and Go HTTP client strings instead. An allowlist of names would never have matched it, and neither would a blocklist.

Training is the one row where blocking is a reasonable choice, and it is a content decision rather than a security one. You may want to allow retrieval and refuse training, or the reverse. Either way, it is not a reason to treat every automated client as suspicious activity.

Can AI Crawlers Execute JavaScript?

Mostly not, which surprises people. In Vercel's analysis of crawler traffic on its network, GPTBot, ChatGPT-User, ClaudeBot, PerplexityBot, and the Meta and ByteDance crawlers fetched HTML, and sometimes downloaded JavaScript files, without executing any of it. The exceptions were Gemini, which renders through Googlebot's infrastructure, and Applebot. Agent browsers are the other exception: ChatGPT's Cloud browser and its peers drive a full browser, so they run everything.

Two practical consequences follow. If your pricing or docs only appear after client-side rendering, most AI crawlers can't see them, and that is a content problem, not a bot problem. And JavaScript execution makes a poor on/off switch for policy. A rule that blocks clients that skip JavaScript hits the polite fetchers. A rule that blocks automated browsers hits agents acting for real customers. The well-built signup bot gets through both, because it renders the page the way any modern form expects.

There is a third consequence for account scoring. The Dregs tracker runs in the browser, and an identity only exists in Dregs once your application identifies a user at signup or sign-in. A crawler that fetches HTML and leaves never becomes an account, so there is nothing to judge and nothing to get wrong. Dregs doesn't police the fetch. It judges what signs up. When a client does run script and register, the evidence gets richer: a registration form filled in 400 milliseconds, followed by the next account from a sibling device, is not a fetcher. Dregs scores that from device rendering, automation signatures, and timing, with confidence attached, so a thin first hit is not treated like a completed campaign.

Allowlists Rot; Scoring Scales

Vendor allowlists and "AI bot" blocklists ask whether a request matches a name you already wrote down. New assistants appear faster than those lists are updated, names change, and the afternoon a list ships, a signup script copies the header.

Verification is getting better, and you should use it. Published IP ranges and reverse DNS have been around for years, and signed requests under Web Bot Auth let an agent prove where it came from cryptographically. ChatGPT's Cloud browser signs its requests, and edge providers such as Cloudflare can check the signature for you. But verification tells you who a client is. It doesn't tell you whether the account it just created is a problem.

Edge bot management still belongs in front of the app for noisy scrapers and for labeling known crawlers. It won't tell you that the "assistant" which just registered used a throwaway inbox, is the fourth account on the same hardware this week, and went straight for the free credits. That is an account question, and identity scoring is how you answer it.

A Humanity and Behavior Policy

Encode the split in custom rules and lists rather than in a global "block bots" toggle. The useful pattern is:

• Automation that never signs up. Fetchers, indexers, monitors, and link previews. Allow them at the edge, set your training preference in robots.txt, and move on. They never become accounts, so there is nothing to score.
• Low Humanity plus registrations, logins, or trial cycling. A claimed assistant that then signs up. Treat it as fake signups, free trial abuse, or credential stuffing, depending on the path. Stop it before it settles into the user table.
• Low Humanity on a paying customer's account, ordinary everything else. An agent doing a job for someone who pays you: pulling a report, filing a ticket, finishing a setup step. Expect more of this every quarter. Behavior, Authenticity, and Uniqueness are what tell you the account is fine, and they should outvote Humanity here.
• High Humanity, low Behavior. A person driving a routine: the trial cycler, the list tester. Humanity won't save you here. Behavior will.
• Automation you operate. QA bots, partner integrations, invited agents. Mark them disregarded so they stop affecting everyone else's analysis. Don't teach the rest of the product that "low Humanity" means "ban."

Dregs scores every account on Humanity, Behavior, Authenticity, and Uniqueness, updated moments after new activity, and each score opens into its observations so you can see exactly why before you act. Real users stay clear of false positives, including the ones whose assistant is doing the clicking, and there is less manual review to do.

Keep Humanity and Behavior as separate knobs. Badge likely bots, escalate only when several signals agree, and let webhooks refuse to provision, throttle, or simply watch. Dregs doesn't replace an edge WAF or a verified-bot catalog. It covers the part they can't see, which is what an account does after the request was allowed.

llms.txt and Crawler Norms

Crawler norms are useful and incomplete. robots.txt is how well-behaved clients ask what they may fetch, and it is also where the training decision lives: GPTBot, ClaudeBot, and CCBot are the crawler names to address, while Google-Extended and Applebot-Extended are tokens that opt your content out of AI training without touching search. Some sites also publish llms.txt at the root, a Markdown summary so assistants can find a clean description of the product.

Both are courtesy. Neither authenticates the client. Neither notices that the same user agent later opened twenty accounts. Honor them at the edge for clients that honor them, and don't confuse a file on disk with bot management for AI crawlers. Abusive scripts ignore the file. Judgment still sits on Humanity, Behavior, and the identity in front of you.

Frequently Asked Questions

Further Reading