Baiduspider reads robots.txt, and Baidu’s handling has its own quirks alongside the familiar directives.
What to know
- Baiduspider has multiple user agents, including separate mobile and image crawlers.
- Directives that other engines ignore may be honoured, and vice versa; do not copy a robots.txt between engines and assume equivalence.
- Blocking Baiduspider entirely is a common accident on sites configured for Western traffic only.
The universal rule
robots.txt controls crawling, not indexing. A blocked URL can still be listed.
Alien Road
How we apply this
We check for an inherited rule blocking Baiduspider on every China project. It is often there deliberately, added years earlier to reduce server load, and nobody remembers it when the business decides to enter the market.
Related services