Robots.txt analysis

The robots.txt analyser does three jobs: validate the file, test specific URLs against it, and keep a change history. The third is the one nobody uses and the one that resolves incidents fastest.

Validation

For a verified site the file appears automatically; otherwise enter the address. Sections addressed to Yandex (User-agent: Yandex oder *) are checked against Yandex’s own rules, and other sections against the general standard — which is why a file that passes a generic checker can still fail here.

Results come as warnings (deviations the tool can correct itself, or possible typos) and errors (a line, section or whole file that cannot be processed).

Testing URLs — the part that matters

The Check if URLs are allowed section takes a list of addresses, full or relative, and marks each as permitted or blocked. This is the only reliable way to know what the file does, because it applies the real precedence rules — a Yandex block overriding the wildcard, a specific robot block overriding both.

Test both directions: URLs that must be crawlable, and URLs that must not be. A syntactically perfect file blocking the wrong directory reports no errors at all.

Change history

Six months of history, up to 100 saved versions, downloadable. When indexing changes suddenly, the first question is whether this file changed — and this answers it with a date, without needing access to anyone’s deployment log. Combined with the “robots.txt modified” notification, it turns an invisible category of incident into a traceable one.

Working practice

Draft in the tool, validate, test the URL list, then copy to the site. Editing the live file and checking afterwards is how sites spend a fortnight blocked — and the file is fetched several times a day, so a mistake propagates quickly.