Crawl statistics is the log of what the robot actually did: which pages it fetched, when, and what the server answered. It is the first report to open when anything about indexing looks wrong, because it describes events rather than states. Data updates daily, within six hours of the robot’s visit, and the history goes back to 20 February 2017.
Two views
- Recent changes — pages whose HTTP response code changed between visits, up to 50,000 entries. Note the limitation carefully: a change is recorded only when the response code changes. Content changing with a steady
200 OKdoes not appear here at all. - All pages — everything the robot has crawled, also capped at 50,000, with crawl date, path and last response code.
Both can be filtered by date, URL fragment or response code and exported to XLS or CSV.
How to read the Was and Currently columns
A first-time crawl shows N/a in Was and the response in Currently; once loaded into the database, the page can appear in results at the next index update. A re-crawl shows the previous response against the current one, which is where the useful patterns are: 200 becoming 404 on a set of URLs is a deployment that broke something, and it is visible here days before it shows up in traffic.
Why deleted pages keep being crawled
Yandex explains a behaviour that puzzles most site owners: a page excluded from search stays in the database while the robot keeps checking whether it has returned. It goes on requesting it as long as links to it exist and robots.txt does not forbid it. So URLs deleted years ago continue to appear in this report — the robot found a link on someone else’s site. If the crawling is genuinely unwanted, a Disallow rule is the remedy; if the URLs are merely old, they cost nothing but noise.
Section-level analysis
The report can be limited to a site section, using the structure Yandex has derived — which is itself informative, since a section the console does not know about is a section the robot has not properly explored. Pages missing from the list entirely should be submitted through the reindexing tool.
The number worth watching
Crawl frequency by section. If the robot visits the blog daily and a commercially important category monthly, that is a crawl-budget finding, and it explains why updates to that category take weeks to have any effect.
Как мы это применяем
This is the first report we open when a client says pages are “not being indexed”, because it separates two completely different problems: the robot never came, or the robot came and the server answered badly. Filtering by response code across a date range takes a minute and settles it. The behaviour we most often have to explain is the robot repeatedly requesting URLs deleted long ago — it is following links from other sites, not malfunctioning.
Связанные услуги