{"id":25712,"date":"2026-09-06T00:50:35","date_gmt":"2026-09-05T21:50:35","guid":{"rendered":"https:\/\/alienroad.com\/google-bilgi-bankasi\/authenticate-requests-with-web-bot-auth-experimental\/"},"modified":"2026-09-06T00:50:35","modified_gmt":"2026-09-05T21:50:35","slug":"authenticate-requests-with-web-bot-auth-experimental","status":"publish","type":"ar_kb","link":"https:\/\/alienroad.com\/google-bilgi-bankasi\/authenticate-requests-with-web-bot-auth-experimental\/","title":{"rendered":"Authenticate requests with Web Bot Auth (experimental)"},"content":{"rendered":"<p>\n  Google is testing usage of the <a class=\"external-link\" href=\"https:\/\/datatracker.ietf.org\/doc\/draft-meunier-http-message-signatures-directory\/\">Web Bot Auth<\/a><br \/>\n  IETF internet draft, which is a new cryptographic protocol that helps websites to validate that<br \/>\n  bots are authentic. We are testing the protocol with some AI agents hosted on Google<br \/>\n  infrastructure. This guide explains what Web Bot Auth is, the current status, and how you can<br \/>\n  implement verification in the experimental phase.\n<\/p>\n<aside class=\"caution\">We don&#8217;t sign every request of a particular agent. Be sure that you fall back<br \/>\n  to the <a href=\"https:\/\/alienroad.com\/google-bilgi-bankasi\/verify-requests-from-google-crawlers-and-fetchers\/\">established methods<br \/>\n  of bot verification<\/a>.<\/aside>\n<h2 id=\"what-is-web-bot-auth\" tabindex=\"-1\">What is Web Bot Auth?<\/h2>\n<p>\n  Web Bot Auth is an experimental cryptographic protocol used to authenticate requests sent by bots.<br \/>\n  Instead of relying solely on self-reported headers and IP addresses, Web Bot Auth allows agents<br \/>\n  to cryptographically sign their requests.\n<\/p>\n<p>\n  Using Web Bot Auth helps website owners identify automated traffic on their sites, and prevents<br \/>\n  other actors from attempting to spoof reputable agents. Web Bot Auth can bring the following benefits:\n<\/p>\n<ul>\n<li><strong>Cryptographic certainty:<\/strong> Move beyond easily spoofed headers to a verified<br \/>\n    identity and decouple agent identity from IP addresses.<\/li>\n<li><strong>Better observability:<\/strong> Gain clearer insights into how agents interact with<br \/>\n    your content.<\/li>\n<li><strong>Future-proofing:<\/strong> Help establish a web where agent providers and websites can<br \/>\n    build mutual trust and make informed access decisions.<\/li>\n<\/ul>\n<h2 id=\"current-status\" tabindex=\"-1\">Current status of Web Bot Auth: experimental<\/h2>\n<p>\n  Google&#8217;s implementation of <a class=\"external-link\" href=\"https:\/\/datatracker.ietf.org\/doc\/draft-meunier-web-bot-auth-architecture\/\">Web Bot Auth<\/a> is currently <em>experimental<\/em><br \/>\n  for the following reasons:\n<\/p>\n<ul>\n<li>Web Bot Auth is currently a draft specification developed by the IETF<br \/>\n    <a class=\"external-link\" href=\"https:\/\/datatracker.ietf.org\/wg\/webbotauth\/about\/\">WBA Working Group<\/a>,<br \/>\n    and it may change over time. Google continues to be engaged with the work of the Working Group<br \/>\n    as it evolves.<\/li>\n<li>User-agent and IP-based bot verification is currently the de facto standard, with decades of<br \/>\n    systems, policies, and best practices built around it. Changing these requires time and a<br \/>\n    careful approach, and we&#8217;re at an early stage, still evaluating the technical characteristics<br \/>\n    and potential ecosystem implications of the protocol.<\/li>\n<\/ul>\n<h3 id=\"what-does-this-mean\" tabindex=\"-1\">What does this mean?<\/h3>\n<p>\nThe experimental status means that:\n<\/p>\n<ul>\n<li>Not all Google user agents are using Web Bot Auth.<\/li>\n<li>Google is not yet signing every request of agents using the protocol.<\/li>\n<li>We recommend that in addition to Web Bot Auth you continue relying on <a href=\"https:\/\/alienroad.com\/google-bilgi-bankasi\/verify-requests-from-google-crawlers-and-fetchers\/\">IP addresses, reverse DNS, and user-agent strings<\/a><br \/>\n    as we gradually roll out signed traffic.<\/li>\n<\/ul>\n<p>\n  For those that are interested in participating during the experimental phase, we&#8217;re providing<br \/>\n  guidance on how to recognize and allowlist Google AI agents.\n<\/p>\n<h2 id=\"how-to-allowlist-google-ai-agents-using-web-bot-auth\" tabindex=\"-1\">How to allowlist Google AI agents using Web Bot Auth<\/h2>\n<p>\n  If you&#8217;re a developer or system administrator looking to allowlist our experimental AI agents, you<br \/>\n  can implement verification through the Web Bot Auth protocol:\n<\/p>\n<ul>\n<li><a href=\"#products-and-services\">Using a product or service that supports Web Bot Auth<\/a><\/li>\n<li><a href=\"#verify-requests-yourself\">Verifying requests yourself<\/a><\/li>\n<\/ul>\n<h3 id=\"products-and-services\" tabindex=\"-1\">Use a product or service that supports Web Both Auth<\/h3>\n<p>\n  Major bot-detection services, CDNs, and WAFs support Web Bot Auth. Some infrastructure services<br \/>\n  provide ways to look up the <code><a href=\"https:\/\/alienroad.com\/google-bilgi-bankasi\/list-of-google-user-triggered-fetchers\/#google-agent\">Google-Agent<\/a><\/code><br \/>\n  user agent and allowlist it; refer to your provider for the exact steps. A <strong>subset<\/strong><br \/>\n  of requests made by the <code>Google-Agent<\/code> are signed with Web Bot Auth; in these<br \/>\n  cases, they are authenticated as <code>https:\/\/agent.bot.goog<\/code>. If your provider supports<br \/>\n  the protocol, they likely verify this automatically.\n<\/p>\n<h3 id=\"verify-requests-yourself\" tabindex=\"-1\">Verify requests yourself<\/h3>\n<p>\n  If you want to authenticate our requests yourself, refer to the<br \/>\n  <a class=\"external-link\" href=\"https:\/\/datatracker.ietf.org\/doc\/draft-meunier-web-bot-auth-architecture\/\">HTTP Message Signatures for automated traffic architecture specification<\/a><br \/>\n  and <a class=\"external-link\" href=\"https:\/\/github.com\/cloudflare\/web-bot-auth\">example implementations on GitHub<\/a>.<br \/>\n  In general, the key protocol steps are:\n<\/p>\n<ol>\n<li>Fetch the public key set of our agent from <a class=\"external-link\" href=\"https:\/\/agent.bot.goog\/.well-known\/http-message-signatures-directory\">https:\/\/agent.bot.goog\/.well-known\/http-message-signatures-directory<\/a><br \/>\n    and cache them according to the <code><a class=\"external-link\" href=\"https:\/\/developer.mozilla.org\/en-US\/docs\/Web\/HTTP\/Reference\/Headers\/Cache-Control\">Cache-Control<\/a><\/code> header.<\/p>\n<aside class=\"note\">Be sure to delete old cached keys that are missing from the file, as they<br \/>\n    might be expired or revoked.<\/aside>\n<\/li>\n<li>Participating requests sent to your server will have the <code>Signature-Agent<\/code> HTTP<br \/>\n    request header set to <code>g=\"https:\/\/agent.bot.goog\"<\/code> (take note of the <code>g=<\/code> label).<\/li>\n<li>Verify the <code>Signature<\/code> header according to the <code>Signature-Input<\/code>, following<br \/>\n    the HTTP Message Signatures standard (<a class=\"external-link\" href=\"https:\/\/datatracker.ietf.org\/doc\/html\/rfc9421\">RFC 9421<\/a>).<br \/>\n    Use the <code>Signature<\/code> and <code>Signature-Input<\/code> headers labeled as <code>g<\/code>.<\/li>\n<li>Don&#8217;t forget to fall back to <a href=\"https:\/\/alienroad.com\/google-bilgi-bankasi\/verify-requests-from-google-crawlers-and-fetchers\/\">IP-based verification<\/a>,<br \/>\n    as not all requests are signed.<\/li>\n<\/ol>\n<p><img decoding=\"async\" src=\"https:\/\/alienroad.com\/wp-content\/uploads\/kb-gorsel\/g-b77fdc46e873.png\" loading=\"lazy\" alt=\"Web Bot Auth verification steps\"><\/p>\n<p>\n  For latency-sensitive requests, you can return the response in advance and validate the signature<br \/>\n  within its <a class=\"external-link\" href=\"https:\/\/datatracker.ietf.org\/doc\/html\/rfc9421#section-2.3-4.4\">expiry window<\/a>.<br \/>\n  In this case, sanctioning will occur after the fact and you can apply that to the caller&#8217;s future requests.\n<\/p>\n<aside class=\"note\">A signature&#8217;s expiry window isn&#8217;t the same as the public key set&#8217;s <code>Cache-Control<\/code><br \/>\n  header. Be sure to validate them separately.<br \/>\n<\/aside>\n<h2 id=\"next-steps\" tabindex=\"-1\">Next steps<\/h2>\n<ul>\n<li>Contact your hosting or security provider to see if they support Web Bot Auth verification.<\/li>\n<li>Stay updated on the technical specifications of the <a class=\"external-link\" href=\"https:\/\/datatracker.ietf.org\/wg\/webbotauth\/about\/\">Web Bot Auth Working Group<\/a>.<\/li>\n<li>Participate in the discussion at the <a class=\"external-link\" href=\"https:\/\/mailman3.ietf.org\/mailman3\/lists\/web-bot-auth@ietf.org\/\">IETF mailing list<\/a>.<\/li>\n<li>Send us feedback through our <a class=\"external-link\" href=\"https:\/\/forms.gle\/HyxC6SdPaAJCDHfv7\">Web Bot Auth feedback form<\/a>.<\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"<p>Learn what Web Bot Auth is, the current status of Google&#8217;s implementation, and how you can implement verification during Google&#8217;s experimental phase.<\/p>\n","protected":false},"menu_order":7000,"template":"","meta":{"footnotes":""},"ar_kb_kategori":[708],"ar_kb_etiket":[],"class_list":["post-25712","ar_kb","type-ar_kb","status-publish","has-post-thumbnail","hentry","ar_kb_kategori-crawling-infrastructure-crawling-and-indexing"],"_links":{"self":[{"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_kb\/25712","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_kb"}],"about":[{"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/types\/ar_kb"}],"version-history":[{"count":0,"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_kb\/25712\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/media\/27677"}],"wp:attachment":[{"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/media?parent=25712"}],"wp:term":[{"taxonomy":"ar_kb_kategori","embeddable":true,"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_kb_kategori?post=25712"},{"taxonomy":"ar_kb_etiket","embeddable":true,"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_kb_etiket?post=25712"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}