Ana Fernández / SEO

Google-Agent: what it is and how to let it into your site

What Google-Agent is, how to identify it (user agent, IP and Web Bot Auth), why it ignores robots.txt and how to allow it in your CDN or your firewall.

November 20, 2026 11 min readby Ana Fernández

Google-Agent is the name Google uses for the agents that browse the web and do things on a site when a user asks them to, and since November 17 it has an official guide so your CDN and your firewall let it through. The guide is short, technical and in English, so at your company someone in IT has probably read it (if anyone has).

A made-up example, to make it concrete. Someone asks a Google assistant to quote travel insurance for their family. The agent lands on your site, reaches the quote tool, fills in the first passenger's age and runs into a CAPTCHA. That's where the visit ends. Your marketing team never finds out, because the form was never submitted and nobody in marketing looks at the CDN's security events.

My take: for a company in Chile or Peru, whether an agent can quote, book or buy on your site is decided by the CDN and firewall rules, which IT or security usually manages. That's why this review comes before any "optimize for agents" project. Here's what Google-Agent is, what Google published, how to recognize it, how to let it in and what I'd check this week.

What is Google-Agent?

According to Google's documentation, Google-Agent is the name used by agents hosted on Google infrastructure to navigate the web and perform actions upon user request. In other words, it shows up when a person hands a task to a Google AI and that AI has to go to your site to do it.

Watch out for the name confusion. If you search "google agent" on Google, the AI summary talks about Gemini Agent and the platforms for building AI agents on Google Cloud. Google-Agent, with a hyphen, is how the agent visiting your site introduces itself to your server. It isn't Googlebot either, the search crawler (more on the difference below).

Google files it under its user-triggered fetchers, the same family as the link previews in Google Messages. It's the only one on that list that the documentation describes as taking actions; the rest handle a single request and read one URL.

Which Google products use it?

Google doesn't say which products send these visits. When it added the agent to its list, on March 20, 2026, it gave Project Mariner as the example, its experimental agent that drives a browser and fills in forms. The new guide doesn't name any product.

The closest volume figure comes from HUMAN, a security company: in October it measured that something it called "Google Agent (Gemini)" generated 0.8% of the agent requests it observed. HUMAN didn't link it to this user agent, so I'd treat it as a hint and nothing more.

What did Google publish on November 17?

On November 17, 2026, Google published a page called "Allow Google-Agent traffic" in its crawling documentation. The changelog describes it as a technical guide to make sure Google-Agent has access to your site. It has three parts: how the agent identifies itself, how to allow it in a CDN, a web application firewall (WAF) or a bot-detection service, and what to check if it's still blocked.

What's new is the direction of the document. Until now Google explained what Google-Agent was; this guide asks the site to change its configuration so the agent works. And the reason it gives is concrete: explicitly letting it through keeps your CDN or your firewall from blocking legitimate requests, without dismantling your security controls.

How do you identify a Google-Agent request?

Every request carries two signals and, sometimes, a third:

SignalWhat it isHow reliable it is
User agentThe request header contains "Google-Agent" (there's a mobile and a desktop version, with a Chrome version that changes)Not much on its own: anyone can copy it
IP addressThe IP appears in the user-triggered-agents.json file Google publishesHigh, as long as your list is up to date
SignatureSome requests come signed with Web Bot Auth, under the agent.bot.goog identityThe strongest, but it's experimental and not every request has it

The user-triggered-agents.json file is small: the November 16 version had 20 IP ranges (12 IPv4 and 8 IPv6), according to PPC Land. Since Google refreshes these files daily, a copy pasted by hand into the firewall goes stale fast.

The signature uses the HTTP Message Signatures standard, RFC 9421, and the public keys to verify it are published at agent.bot.goog. Google says only part of Google-Agent's requests are signed, so IP address verification is still required. A practical detail for whoever writes the rule: in the desktop version, the user agent places "Google-Agent" before the Chrome version rather than at the end, so it's safer to check that the header contains it anywhere.

Does Google-Agent respect robots.txt?

Generally, no. Google says user-triggered fetchers generally ignore robots.txt rules, because a person requested the visit. The November 17 guide doesn't even mention robots.txt.

That changes where the decision gets made. If someone at your company blocked or allowed AI bots in robots.txt (as I explained in this guide), that rule probably doesn't touch Google-Agent. Googlebot does respect robots.txt, but the agent acting for a user is controlled at another layer: the CDN, the firewall and the bot filters.

How is it different from Googlebot?

GooglebotGoogle-Agent
Why it visitsCrawl and index for search, AI Overviews and AI ModeDo a task a person asked for
Robots.txtRespects itGenerally ignores it
Where you control itrobots.txt and meta tagsCDN, firewall and bot-detection services
What it does on your siteReads pagesSearches, filters, opens detail pages and can move through a purchase or a booking

Search and AI Mode are fed by Googlebot, so blocking or allowing Google-Agent shouldn't affect your indexing. Google says nothing about any effect on rankings, and I wouldn't assume one.

Why does this matter to a marketing team?

Because the list of problems Google published describes a purchase journey. It mentions on-site search, filters, product pages, add to cart, booking and the API calls behind every button. That's exactly the path a marketing team spends years optimizing so a person converts.

And security defenses judge automation by how it looks: traffic from a data center, fast navigation, nobody moving the mouse. An agent that's there to buy looks a lot like one that's there to scrape prices. That's why blocks happen without anyone deciding them on purpose, and without leaving a trace in any log marketing reviews.

There's some third-party data, worth reading for what it is (vendor studies). IWD, a B2B ecommerce agency, reviewed 99 sites: only 2 of 89 robots.txt files blocked AI on purpose, but 12 sites rejected, at the firewall or CDN, a request that identified itself as an AI bot. Blend, which tests a shopping agent, reported that bot protection blocked it at 29 of 200 retailers in the UK and the US.

In Latin America the pattern shows up another way. As I wrote in robots.txt and AI training, since September 15 Cloudflare blocks training and agent bots by default on pages with ads on new domains. What I've seen on large sites is that the SEO team reviews robots.txt and security reviews the WAF, and those two conversations very rarely happen in the same meeting.

How do you allow Google-Agent in your CDN or firewall?

Start by asking your provider. Google says some bot-detection services, CDNs and firewalls already verify Google-Agent and let it through with a setting; the exact steps depend on each one. If yours doesn't, the guide lays out four rules in this order:

  1. Check the user agent. Confirm the User-Agent header contains "Google-Agent".

  2. Check the IP. Confirm the address comes from a range published in user-triggered-agents.json.

  3. Check the signature, if there is one. Verify the Signature and Signature-Input headers with Web Bot Auth for the agent.bot.goog identity.

  4. Skip the challenges. Don't show verified requests a CAPTCHA, JavaScript challenges or interstitial pages.

Step 4 is the one that actually changes something, because the CAPTCHA is the main tool against automated traffic. And a warning: an allow rule based only on the user agent opens the door to anyone who copies it. The three checks go together, and if your provider has a log-only mode, it's worth testing the rule there before switching it on.

Why is Google-Agent still blocked after you allow it?

This is the most useful part of the guide. Google lists six reasons a legitimate request can keep bouncing, and I'll translate them into what they mean on a site with a quote tool or a shopping cart:

CheckWhat Google asksWhat it looks like on your site
Evaluation orderThe allow rule takes priority and runs before data center blocks, managed filters and "block AI agents" categoriesYou turned on the allow rule, but a broader block runs first and wins
Interactive challengesSkip CAPTCHAs and JavaScript challenges, including the ones that appear on search or form submissionThe home page loads and the quote tool asks for a challenge
In-journey actionsThe allow covers on-site search, filters, detail pages, cart, bookings and the XHR, GraphQL or api.* endpointsThe agent sees the category, but the cart API returns a 403
Per-IP limitsPer-IP limits account for many people leaving through the same pool of addressesTen users with their agents look like one very insistent IP
Outdated rangesAllowed ranges stay synced with the latest user-triggered-agents.jsonSomeone copied the ranges in March and never touched them again
Proxy headersThe load balancer or proxy keeps the original IP address, the user agent and each request's signatureThe firewall can't verify anything because it receives the load balancer's IP

The fourth row is usually the hardest. Per-IP limits are one of the oldest defenses on the web, and exempting Google's ranges means trusting Google's own traffic controls in their place.

Should you let it into your site?

It depends on what you want an agent to be able to do for your customer. In the robots.txt article I laid out two separate decisions for AI bots: whether your content trains models and whether you want to be cited. Google-Agent adds a third: whether you want an agent to be able to act on your site.

Type of siteWhat I'd do
EcommerceAllow it on search, categories, product pages and cart; that's what agentic shopping projects prepare for
Insurance and fintech with a quote toolAllow it up to the quote result and decide with security and legal what happens at purchase
Travel, hotels and services with bookingsAllow it on availability and booking, with per-IP limits designed for shared traffic
B2B SaaSAllow it on pricing, documentation and demo forms
Media or publisher with a paywallReview it with the subscriptions team, because the agent may reach paid content if your setup lets it

If you're already looking at protocols like UCP or WebMCP, this review comes first: there's little point describing your catalog to an agent if the firewall won't let it past the category page.

What doesn't Google's guide say yet?

Worth having clear before the meeting with IT:

  • It gives no volume: how many visits Google-Agent makes or which countries it operates in.
  • It doesn't say what share of requests is signed with Web Bot Auth.
  • It doesn't name the products that use it.
  • It doesn't explain how to block it, only how to allow it.
  • It doesn't say whether it will show up with its own label in analytics or whether it has any effect on rankings.

How do you check whether your site lets Google-Agent through?

This is what I'd do this week, with no extra budget:

  1. Ask whoever manages the CDN three questions. Do we verify Google-Agent or treat it as an unknown bot? Do our challenge rules exclude it? Does the allow rule cover search, filters, cart, quote tool and the API?

  2. Search your logs for "Google-Agent". Check that those visits get a normal page (status 200) and not a 403 or a challenge. The step-by-step for grouping bots is in log file analysis for SEO.

  3. Look at the CDN log too. If the CDN blocks the visit before it reaches your server, it won't show up in the origin log. You need to check the provider's security events.

  4. Check other bots while you're at it. OAI-SearchBot, ChatGPT's search bot, and PerplexityBot can fall under the same rules. None of them shows up in Search Console, so logs are the only way to see them.

If you then find that the agent bounces off your quote tool, you have a concrete problem to prioritize with IT, measured on the journey that matters most to the business. And if everything gets through cleanly, you can move on to content and protocols knowing the agent makes it to the end of the journey.

Free weekly resource

Get the enterprise SEO playbooks before anyone else

Every week we publish a new free resource: interactive audits, calculators, checklists and templates we actually use with enterprise clients. Leave your email and we'll send it to you first.

No spam. Unsubscribe whenever you want.

Enterprise SEO consulting

Want to apply this to your brand?

We design end-to-end SEO and GEO programs connected to pipeline and revenue. Senior team, weekly execution, executive reporting.