Most sites contain pages nobody should find in search: service and administration pages, content useful only to logged-in users such as checkout screens and profiles, drafts and placeholders. Yandex explains why hiding them is not housekeeping but a ranking concern.
The three costs of leaving them crawlable
- Unnecessary load on the site from crawling them.
- The robot takes longer to reach your landing pages — crawl capacity spent on pages that will never rank.
- If they are indexed, they compete with your important pages and confuse users.
Yandex adds the observation that makes this urgent on large sites: the number of such pages is comparable to, and often exceeds, the number of pages worth attracting visitors to.
Finding out which ones are actually being crawled
Searchable pages → Excluded pages, filtered to low-value or low-demand. Yandex points out that this list contains two different things:
- Pages you do want in search, which need improving.
- Pages never intended for users, which the robot found anyway — and those should be hidden.
Reading the list with that split in mind turns an anxiety-inducing report into two concrete work items.
How to hide them
The instrument depends on the goal. Disallow in robots.txt stops crawling and saves capacity. noindex keeps the page out of results while allowing the robot to read it. Authorisation removes the question entirely for pages that should never be public. The two must not be combined on the same URL — a blocked page cannot be read, so its noindex is never seen.
The practical priority
On a catalogue site, the highest-value exclusions are filter and sort combinations, internal search results, and cart and comparison pages. That set is usually where the crawl budget is going, and blocking it is a single afternoon’s work with a measurable effect on how quickly new products appear.
Comment nous l'appliquons
The two-lists reading of the low-value report is how we run this: half the entries are usually pages the client wants and needs to improve, the other half are pages nobody meant to publish. Separating them takes twenty minutes and produces a content plan and a robots.txt change instead of a vague worry about a large number.
Services associés