{"id":28221,"date":"2026-09-08T21:52:59","date_gmt":"2026-09-08T18:52:59","guid":{"rendered":"https:\/\/alienroad.com\/baidu-bilgi-bankasi\/robots-txt-and-baiduspider\/"},"modified":"2026-09-09T02:55:31","modified_gmt":"2026-09-08T23:55:31","slug":"robots-txt-and-baiduspider","status":"publish","type":"ar_bkb","link":"https:\/\/alienroad.com\/baidu-bilgi-bankasi\/robots-txt-and-baiduspider\/","title":{"rendered":"robots.txt and Baiduspider"},"content":{"rendered":"

Baiduspider checks robots.txt<\/code> in the root of the domain before crawling. The syntax follows the standard protocol; the details below are where Baidu differs or is more explicit than most engines.<\/p>\n

The rule most sites get wrong<\/h2>\n

Baidu states it plainly: if you want everything indexed, do not create a robots.txt file at all.<\/strong> The file exists only to exclude. An empty or absent file leaves the site fully open, which is the correct configuration for most sites and the one people replace with a half-remembered template.<\/p>\n

Blocking is not removal<\/h2>\n

A page blocked in robots.txt<\/code> can still appear in Baidu results<\/strong> when other sites link to it. Its content is not crawled, indexed or displayed \u2014 what appears is other sites’ description of it. Blocking controls crawling; it does not control presence.<\/p>\n

Directive precision<\/h2>\n
    \n
  • Disallow: \/help<\/code> blocks \/help.html<\/code>, \/helpabc.html<\/code> and<\/em> \/help\/index.html<\/code>.<\/li>\n
  • Disallow: \/help\/<\/code> blocks only \/help\/index.html<\/code> \u2014 the first two remain crawlable.<\/li>\n
  • An empty Disallow:<\/code> permits everything; at least one Disallow<\/code> record is required in a valid file.<\/li>\n
  • Order matters:<\/strong> the robot applies the first matching Allow<\/code> v\u0259 ya Disallow<\/code> line<\/em>. This is not the longest-match rule other engines use, and a file written for Google can behave differently here.<\/li>\n
  • Wildcards: *<\/code> matches any sequence, $<\/code> matches end of line.<\/li>\n
  • Matching is case-sensitive and exact.<\/strong> Baidu warns that a case mismatch makes the rule ineffective.<\/li>\n<\/ul>\n

    Baidu-specific meta directives<\/h2>\n
      \n
    • <meta name=\"robots\" content=\"nofollow\"><\/code> \u2014 do not follow links or pass value; <meta name=\"Baiduspider\" content=\"nofollow\"><\/code> restricts this to Baidu alone.<\/li>\n
    • <meta name=\"robots\" content=\"noarchive\"><\/code> \u2014 no cached snapshot; the Baidu-only form is <meta name=\"Baiduspider\" content=\"noarchive\"><\/code>. Note that noarchive suppresses the snapshot only<\/strong> \u2014 the page is still indexed and still shown with a summary.<\/li>\n
    • Per-link control with rel=\"nofollow\"<\/code> is supported.<\/li>\n<\/ul>\n

      Identifying the real Baiduspider<\/h2>\n

      Baidu does not publish its IP ranges; they change. Verification is by user agent plus reverse DNS, and there are three user-agent families \u2014 mobile<\/strong>, PC<\/strong>, and mini-program<\/strong>, with Baiduspider-render<\/code> variants for rendering. Blocking by IP is not a supported strategy; checking the agent first and confirming by DNS is.<\/p>\n","protected":false},"excerpt":{"rendered":"

      Baiduspider checks robots.txt in the root of the domain before crawling. The syntax follows the standard protocol; the details below are where Baidu differs or is more explicit than most engines. The rule most sites get wrong Baidu states it plainly: if you want everything indexed, do not create a robots.txt file at all. The […]<\/p>\n","protected":false},"menu_order":9,"template":"","meta":{"footnotes":""},"ar_bkb_kategori":[744],"ar_bkb_etiket":[],"class_list":["post-28221","ar_bkb","type-ar_bkb","status-publish","has-post-thumbnail","hentry","ar_bkb_kategori-indexing-and-submission"],"_links":{"self":[{"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_bkb\/28221","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_bkb"}],"about":[{"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/types\/ar_bkb"}],"version-history":[{"count":2,"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_bkb\/28221\/revisions"}],"predecessor-version":[{"id":29220,"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_bkb\/28221\/revisions\/29220"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/media\/28985"}],"wp:attachment":[{"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/media?parent=28221"}],"wp:term":[{"taxonomy":"ar_bkb_kategori","embeddable":true,"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_bkb_kategori?post=28221"},{"taxonomy":"ar_bkb_etiket","embeddable":true,"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/ar_bkb_etiket?post=28221"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}