{"id":25687,"date":"2026-09-06T00:48:40","date_gmt":"2026-09-05T21:48:40","guid":{"rendered":"https:\/\/alienroad.com\/google-bilgi-bankasi\/managing-crawling-of-faceted-navigation-urls\/"},"modified":"2026-09-06T00:51:34","modified_gmt":"2026-09-05T21:51:34","slug":"managing-crawling-of-faceted-navigation-urls","status":"publish","type":"ar_kb","link":"https:\/\/alienroad.com\/google-bilgi-bankasi\/managing-crawling-of-faceted-navigation-urls\/","title":{"rendered":"Managing crawling of faceted navigation URLs"},"content":{"rendered":"<p>\n  Faceted navigation is a common feature of websites that allows its visitors to change how items<br \/>\n  (for example, products, articles, or events) are displayed on a page. It&#8217;s a popular and useful<br \/>\n  feature, however its most common implementation, which is based on URL parameters, can generate<br \/>\n  infinite URL spaces which harms the website in a couple ways:\n<\/p>\n<ul>\n<li>\n    <strong>Overcrawling<\/strong>: Because the URLs created for the faceted navigation seem to be<br \/>\n    novel and crawlers can&#8217;t determine whether the URLs are going to be useful without crawling<br \/>\n    first, the crawlers will typically access a very large number of faceted navigation URLs before<br \/>\n    the crawlers&#8217; processes determine the URLs are in fact useless.\n  <\/li>\n<li>\n    <strong>Slower discovery crawls<\/strong>: Stemming from the previous point, if crawling is spent<br \/>\n    on useless URLs, the crawlers have less time to spend on new, useful URLs.\n  <\/li>\n<\/ul>\n<p>\n  A typical faceted navigation URL may contain various parameters in the query string related to the<br \/>\n  properties of items they filter for. For example:\n<\/p>\n<div><\/div>\n<p>\n  Changing any of the URL parameters <code>products<\/code>, <code>color<\/code>, and<br \/>\n  <code>size<\/code> would show a different set of items on the underlying page. This often means a<br \/>\n  very large number of possible combinations of filters, which translates to a very large number of<br \/>\n  possible URLs. To save your resources, we recommend dealing with these URLs one of the following<br \/>\n  ways:\n<\/p>\n<ul>\n<li>\n    If you don&#8217;t need the faceted navigation URLs potentially indexed, prevent crawling of these<br \/>\n    URLs.\n  <\/li>\n<li>\n    If you need the faceted navigation URLs potentially indexed, ensure that the URLs follow our<br \/>\n    best practices outlined in the following section. Keep in mind that crawling faceted URLs tends<br \/>\n    to cost sites large amounts of computing resources due to the sheer amount of URLs and<br \/>\n    operations needed to render those pages.\n  <\/li>\n<\/ul>\n<h2 id=\"prevent-crawling-of-faceted-navigation-urls\" tabindex=\"-1\">Prevent crawling of faceted navigation URLs<\/h2>\n<p>\n  If you want to save server resources and you don&#8217;t need your faceted navigation URLs to show up in<br \/>\n  Google Search or other Google products, you can prevent crawling of these URLs with one of the following ways.\n<\/p>\n<ul>\n<li>\n    <strong>Use <a href=\"https:\/\/alienroad.com\/google-bilgi-bankasi\/how-to-write-and-submit-a-robots-txt-file\/\">robots.txt<\/a> to disallow crawling of faceted navigation URLs.<\/strong> Oftentimes<br \/>\n    there&#8217;s no good reason to allow crawling of filtered items, as it consumes server resources for<br \/>\n    no or negligible benefit; instead, allow crawling of just the individual items&#8217; pages along with<br \/>\n    a dedicated listing page that shows all products without filters applied.<\/p>\n<div><\/div>\n<\/li>\n<li>\n    <strong>Use URL fragments to specify filters.<\/strong><\/p>\n<aside class=\"important\"><a href=\"https:\/\/alienroad.com\/google-bilgi-bankasi\/url-structure\/#fragments\" class=\"external-link\">Google Search generally doesn&#8217;t support URL fragments in crawling and indexing<\/a>.<\/aside>\n<p>    If your filtering mechanism is based on URL fragments, it will have no impact on crawling<br \/>\n    (positive or negative). For example, instead of URL parameters, use URL fragments:<\/p>\n<div><\/div>\n<\/li>\n<\/ul>\n<p>\n  Other ways to signal a preference of which faceted navigation URLs (not) to crawl is using<br \/>\n  <code>rel=\"canonical\"<\/code> <code>link<\/code> element and the <code>rel=\"nofollow\"<\/code> anchor<br \/>\n  attribute. However, these methods are generally less effective in the long term than the<br \/>\n  previously mentioned methods.\n<\/p>\n<ul>\n<li>\n    <strong>Using <a href=\"https:\/\/alienroad.com\/google-bilgi-bankasi\/specify-canonical\/#rel-canonical-link-method\" class=\"external-link\"><code>rel=\"canonical\"<\/code><\/a><br \/>\n      to specify which URL is the canonical version of a faceted navigation URL<\/strong><br \/>\n    may, over time, decrease the crawl volume of non-canonical versions of those URLs. For example,<br \/>\n    if you have 3 filtered page types, consider pointing the <code>rel=\"canonical\"<\/code> to the<br \/>\n    unfiltered version:<br \/>\n    <code>https:\/\/example.com\/items.shtm?products=fish&color=radioactive_green&size=tiny<\/code><br \/>\n    specifies <code>&lt;link rel=\"canonical\" href=\"https:\/\/example.com\/items.shtm?products=fish\" ><\/code>.\n  <\/li>\n<li>\n    <strong>Using<br \/>\n      <a href=\"https:\/\/alienroad.com\/google-bilgi-bankasi\/rel-attributes\/#nofollow\" class=\"external-link\"><code>rel=\"nofollow\"<\/code><\/a><br \/>\n      attributes on anchors pointing to filtered results pages<\/strong><br \/>\n    may be beneficial, however keep in mind that every anchor pointing to a specific URL must have<br \/>\n    the <code>rel=\"nofollow\"<\/code> attribute in order for it to be effective.\n  <\/li>\n<\/ul>\n<h2 id=\"ensure-the-faceted-navigation-urls-are-optimal-for-the-web\" tabindex=\"-1\">Ensure the faceted navigation URLs are optimal for the web<\/h2>\n<p>\n  If you need your faceted navigation URLs to be potentially crawled and indexed, ensure you&#8217;re<br \/>\n  following these best practices to minimize the negative effects of crawling the large number of<br \/>\n  potential URLs on your site:\n<\/p>\n<aside class=\"warning\">\n  Keep in mind that having these URLs crawled means an increased resource usage on your server and,<br \/>\n  potentially, slower discovery of new URLs on your site.<br \/>\n<\/aside>\n<ol>\n<li>\n    <strong>Use the industry standard URL parameter separator &#8216;<code>&<\/code>&#8216;.<\/strong> Characters<br \/>\n    like comma (<code>,<\/code>), semicolon (<code>;<\/code>), and brackets (<code>[<\/code> and<br \/>\n    <code>]<\/code>) are hard for crawlers to detect as parameter separators (because most often<br \/>\n    they&#8217;re not separators).\n  <\/li>\n<li>\n    If you&#8217;re encoding filters in the URL path, such as<br \/>\n    <code>\/products\/<strong>fish<\/strong>\/<strong>green<\/strong>\/<strong>tiny<\/strong><\/code>,<br \/>\n    ensure that the logical order of the filters always stays the same and that no duplicate filters<br \/>\n    can exist.\n  <\/li>\n<li>\n    <strong>Return an HTTP <code>404<\/code> status code when a filter combination doesn&#8217;t return<br \/>\n      results.<\/strong><br \/>\n    If there are no green fish in the site&#8217;s inventory, users as well as crawlers should receive a<br \/>\n    &#8220;not found&#8221; error with the proper HTTP status code (<code>404<\/code>). This should also be the<br \/>\n    case if the URL contains duplicate filters or otherwise nonsensical filter combinations, and<br \/>\n    nonexistent pagination URLs. Similarly, if a filter combination has no results, don&#8217;t redirect<br \/>\n    to a common &#8220;not found&#8221; error page. Instead, serve a &#8220;not found&#8221; error with the <code>404<\/code><br \/>\n    HTTP status code under the URL where it was encountered.<\/p>\n<aside class=\"note\">\n      If<br \/>\n      <a href=\"https:\/\/alienroad.com\/google-bilgi-bankasi\/javascript-seo-basics\/#avoid-soft-404s\" class=\"external-link\">you have a single-page app<\/a> this might not be possible.<br \/>\n      Follow the best practices for single page apps.<br \/>\n    <\/aside>\n<\/li>\n<\/ol>\n","protected":false},"excerpt":{"rendered":"<p>Learn what faceted navigation is and how you can optimize Google&#8217;s crawling of large number of faceted navigation URLs.<\/p>\n","protected":false},"menu_order":7000,"template":"","meta":{"footnotes":""},"ar_kb_kategori":[708],"ar_kb_etiket":[],"class_list":["post-25687","ar_kb","type-ar_kb","status-publish","has-post-thumbnail","hentry","ar_kb_kategori-crawling-infrastructure-crawling-and-indexing"],"_links":{"self":[{"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_kb\/25687","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_kb"}],"about":[{"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/types\/ar_kb"}],"version-history":[{"count":1,"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_kb\/25687\/revisions"}],"predecessor-version":[{"id":25716,"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_kb\/25687\/revisions\/25716"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/media\/27672"}],"wp:attachment":[{"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/media?parent=25687"}],"wp:term":[{"taxonomy":"ar_kb_kategori","embeddable":true,"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_kb_kategori?post=25687"},{"taxonomy":"ar_kb_etiket","embeddable":true,"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_kb_etiket?post=25687"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}